Flat networks: Tailscale

This is part of my endeavour to create a flat network for my home lab, servers and devices.

Tailscale

While researching about all the available software and implementations for setting up my own flat-vpn-mesh kind of network to to talk to my parent's house VMs, my own and all through a nice network while on the go, Tailscale came up right from the beginning.

Tailscale is a solution that uses wireguard behind the scenes, and they offer their co-ordination service to manage everything from their web panel.

Tailscale requires the use of an external ID provider, by default it seems it allows Google and Microsoft accounts.

(this by itself led me to proceed with extreme caution and from start put an expiration stamp on this project for me)

I have not checked whether the client is opensource (I think I read that it is), but since the coordinating happens from a hidden service managed by Tailscale company it does not make much difference anyway, if they want they can easily get in my network.

But for the sake of playing with flat networks I installed it on some servers and devices and surprisingly it works really well.

Before playing with Tailscale I gave nebula a try and I did not manage to reach a machine inside one NAT network from a machine in a different NAT network. So, in frustration I tried Tailscale.

Now, I can reach all machines nicely and I found out that all updates are pushed down to clients, which is nice and simple to manage.

DNS

The other interesting thing Tailscale offers is what they call "Magic DNS". Magic DNS is effectively an internal DNS server they expose on 100.100.100.100 and when enabled it will return the IP of my connected devices when I am requesting them by name. Kind of like DNS-SD I guess.

This makes it really easy to use this considering the IPs your devices get are not memorable but random IPs inside the 100.64.0.0/10 block.

So if my web server's name on Tailscale is "web-server" then I can reach it from another Tailscale connected device like so: curl http://web-server/ and it will work!

The Magic DNS functionality can only be enabled from the webui if I set up a set of Nameservers for my Tailscale network. They can be public DNS like 1.1.1.1 or internal like my Pi-hole.

The requirement comes from the fact that the Magic DNS is only resolving records for my Tailscale devices and nothing more. It will not forward DNS requests for other records. When enabling the Magic DNS, Tailscale will push down to all connected devices a set of resolv.conf entries, one for the search domain and the rest for the nameservers to be used. The first is the Magic DNS and the next ones are the ones I have set up manually for my network. That is why there is a need for a selection of nameservers because otherwise only the Magic DNS will be there and only Tailscale addresses would be resolvable.

But, there seems to be a problem with the client software, that actually when I enable Magic DNS, only the Magic nameserver is added to the resolv.conf of my Linux clients! This is a no-go!

After some playing with it, it appears I can not add the 100.100.100.100 IP as one of my nameservers, due to some validation, to prevent people using it without going through the proper procedure I guess.

Nevertheless, it appears that when Magic DNS is turned off, I can still request records from the 100.100.100.100 nameserver, which is great as now I can set up my stuff as I wish!

From the resolv.conf the search domain seems to be unique to my account (more info on the docs page for Magic DNS) like so: example.gmail.com.beta.tailscale.net. Be careful to get it right, the search domain includes your account's email address with some characters flattened to dots.

CoreDNS to the rescue:

So I thought, I want to use my pihole (which is now listening on a Tailscale IP as well) and be able to have some sort of split-DNS.

Split DNS because I want Pi-hole to return some my already set custom records for the internal network, but I also want to use different records for the same services if I am connected to the Tailscale network. Usually Split DNS is for internal and external networks, but I want to have tailscale-internal-external records.

So the approach is to use CoreDNS which I am using for my OKD lab to answer requests for the Tailscale hostnames and forward to my Pihole the rest.

CoreDNS can be set up to only resolve some domains and forward the rest. I am using Docker to manage it.

The docker-compose.yml file:

version: "3.2"

services:
  coredns:
    image: coredns/coredns
    command: -conf /root/Corefile
    ports:
      - "53:53/udp"
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE
    restart: always
    volumes:
      - ./config:/root/

And the Corefile: (remember to put this and all other zone files in the config directory)

.:53 {
    forward . PIHOLE_IP_NON_TAILSCALE
    errors
}

example.gmail.com.beta.tailscale.net:53 {
    forward . 100.100.100.100
    errors
}

100.64.0.0/10 {
    forward . 100.100.100.100
    errors
}

my.internal.zone.com:53 {
    file /root/my.internal.zone.com.zone
    errors
    reload 60s
}

For the internal zone file, it just needs to be a regular BIND zone file like this.

After starting the CoreDNS container, I added the IP of the host to the nameserver list of my network (also drag it to the top so it is the first one to be queried) and voilà it works for all Tailscale devices!

And when a device in my internal network is not connected to Tailscale then it will get the normal pihole IP from DHCP as before, no harm!

The only drawback with the above implementation is that all requests coming from Tailscale and CoreDNS to Pihole will show as the IP of the CoreDNS host and not the actual machines. Not a big issue though.

Thoughts on Tailscale

As I have mentioned above everything works, but the fact that I can't really trust my network to some other centrally managed service and the fact that auth is managed by Google means I really need to find alternatives and not keep this any longer in my network.

If the control server was open sourced that would be great because then I would be 100% responsible for my network and would not rely on 3rd parties. This issue is tracking open-sourcing the control server but it seems they are not going to do it soon: https://github.com/tailscale/tailscale/issues/498.

In the meantime, I will be following the progress of headscale a 3rd party implementation of the control-server.