Flat networks: Tailscale

This is part of my endeavour to create a flat network for my home lab, servers and devices.

Tailscale

While researching about all the available software and implementations for setting up my own flat-vpn-mesh kind of network to to talk to my parent's house VMs, my own and all through a nice network while on the go, Tailscale came up right from the beginning.

Tailscale is a solution that uses wireguard behind the scenes, and they offer their co-ordination service to manage everything from their web panel.

Tailscale requires the use of an external ID provider, by default it seems it allows Google and Microsoft accounts.

(this by itself led me to proceed with extreme caution and from start put an expiration stamp on this project for me)

I have not checked whether the client is opensource (I think I read that it is), but since the coordinating happens from a hidden service managed by Tailscale company it does not make much difference anyway, if they want they can easily get in my network.

But for the sake of playing with flat networks I installed it on some servers and devices and surprisingly it works really well.

Before playing with Tailscale I gave nebula a try and I did not manage to reach a machine inside one NAT network from a machine in a different NAT network. So, in frustration I tried Tailscale.

Now, I can reach all machines nicely and I found out that all updates are pushed down to clients, which is nice and simple to manage.

DNS

The other interesting thing Tailscale offers is what they call "Magic DNS". Magic DNS is effectively an internal DNS server they expose on 100.100.100.100 and when enabled it will return the IP of my connected devices when I am requesting them by name. Kind of like DNS-SD I guess.

This makes it really easy to use this considering the IPs your devices get are not memorable but random IPs inside the 100.64.0.0/10 block.

So if my web server's name on Tailscale is "web-server" then I can reach it from another Tailscale connected device like so: curl http://web-server/ and it will work!

The Magic DNS functionality can only be enabled from the webui if I set up a set of Nameservers for my Tailscale network. They can be public DNS like 1.1.1.1 or internal like my Pi-hole.

The requirement comes from the fact that the Magic DNS is only resolving records for my Tailscale devices and nothing more. It will not forward DNS requests for other records. When enabling the Magic DNS, Tailscale will push down to all connected devices a set of resolv.conf entries, one for the search domain and the rest for the nameservers to be used. The first is the Magic DNS and the next ones are the ones I have set up manually for my network. That is why there is a need for a selection of nameservers because otherwise only the Magic DNS will be there and only Tailscale addresses would be resolvable.

But, there seems to be a problem with the client software, that actually when I enable Magic DNS, only the Magic nameserver is added to the resolv.conf of my Linux clients! This is a no-go!

After some playing with it, it appears I can not add the 100.100.100.100 IP as one of my nameservers, due to some validation, to prevent people using it without going through the proper procedure I guess.

Nevertheless, it appears that when Magic DNS is turned off, I can still request records from the 100.100.100.100 nameserver, which is great as now I can set up my stuff as I wish!

From the resolv.conf the search domain seems to be unique to my account (more info on the docs page for Magic DNS) like so: example.gmail.com.beta.tailscale.net. Be careful to get it right, the search domain includes your account's email address with some characters flattened to dots.

CoreDNS to the rescue:

So I thought, I want to use my pihole (which is now listening on a Tailscale IP as well) and be able to have some sort of split-DNS.

Split DNS because I want Pi-hole to return some my already set custom records for the internal network, but I also want to use different records for the same services if I am connected to the Tailscale network. Usually Split DNS is for internal and external networks, but I want to have tailscale-internal-external records.

So the approach is to use CoreDNS which I am using for my OKD lab to answer requests for the Tailscale hostnames and forward to my Pihole the rest.

CoreDNS can be set up to only resolve some domains and forward the rest. I am using Docker to manage it.

The docker-compose.yml file:

version: "3.2"

services:
  coredns:
    image: coredns/coredns
    command: -conf /root/Corefile
    ports:
      - "53:53/udp"
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE
    restart: always
    volumes:
      - ./config:/root/

And the Corefile: (remember to put this and all other zone files in the config directory)

.:53 {
    forward . PIHOLE_IP_NON_TAILSCALE
    errors
}

example.gmail.com.beta.tailscale.net:53 {
    forward . 100.100.100.100
    errors
}

100.64.0.0/10 {
    forward . 100.100.100.100
    errors
}

my.internal.zone.com:53 {
    file /root/my.internal.zone.com.zone
    errors
    reload 60s
}

For the internal zone file, it just needs to be a regular BIND zone file like this.

After starting the CoreDNS container, I added the IP of the host to the nameserver list of my network (also drag it to the top so it is the first one to be queried) and voilà it works for all Tailscale devices!

And when a device in my internal network is not connected to Tailscale then it will get the normal pihole IP from DHCP as before, no harm!

The only drawback with the above implementation is that all requests coming from Tailscale and CoreDNS to Pihole will show as the IP of the CoreDNS host and not the actual machines. Not a big issue though.

Thoughts on Tailscale

As I have mentioned above everything works, but the fact that I can't really trust my network to some other centrally managed service and the fact that auth is managed by Google means I really need to find alternatives and not keep this any longer in my network.

If the control server was open sourced that would be great because then I would be 100% responsible for my network and would not rely on 3rd parties. This issue is tracking open-sourcing the control server but it seems they are not going to do it soon: https://github.com/tailscale/tailscale/issues/498.

In the meantime, I will be following the progress of headscale a 3rd party implementation of the control-server.

Diagrams as code using docker + diagram.py

So a colleague showed this python tool to me today and I really liked how easy it is.

Also it has nice icons.

https://github.com/mingrammer/diagrams

In the documentation for diagrams.py the main way to use it is by installing pip and graphviz on my system and the usual python stuff, meaning that either I will have a bloated python system or I have to take care of so many different virtual envs.

These days I try to usually put any tools I want to reuse often into a docker container and run from there.

So for this python tool, I will just make a Dockerfile and build it and then run the python in docker.

Every time I want to run it, it will run from the custom built docker container and that is going to be it, no bloated python setups.

Quite easy with a simple Dockerfile:

FROM python:3
RUN pip install diagrams
RUN apt-get update && apt-get install -y \
    graphviz \
 && rm -rf /var/lib/apt/lists/*

And then just mount the directory the python script is in, and have it run nice and easy like so:

docker run --rm -it -v "${PWD}":/diagram -w /diagram diagrams python diagram.py

The docker command removes the finished container so it is not polluting your docker container cache (the docker image that was built in the first step will remain).

I pushed the Dockerfile and an example here: https://github.com/ledakis/infra-diagrams-python-docker

Simple Fargate project

I got asked to showcase a very simple website that shows something like a page and a picture, that can scale, is built on AWS and with large enterprise in mind.

I have created it on https://github.com/ledakis/simplesite-scale

It includes:

1. Docker container for the app itself, which runs nginx and has the content of the site baked into the container.
2. Terraform for the supporting infrastructure the service will run on.
   • VPC/Subnets using the official terraform module.
   • DNS zone + config for my Cloudflare master zone.
   • ECR for the repository the Docker is going to be hosted in.
   • S3 bucket for the application access logs.
   • ACM certificate for the service.
3. Terraform for the service itself. This deploys:
   • The ALB
   • The ECS service (cluster, task definition, service)
   • Scaling policy (number of connections per target from the ALB)
   • IAM roles

This is a WIP. I want to enhance the scaling, maybe to use the lambda that people generally suggest.

Update 4/12/2019:

It appears EKS+Fargate is a thing now and this sounds very, very interesting!

• AWS Docs on EKS Fargate
• Announcement for the EKS Fargate
• terraform EKS Fargate Profile support

For v2.0 of this project, I will plan to work on the following:

1. Convert the task definition to the k8s manifest and try to have terraform deploy it. This is only to be a test, as it will probably involve spinning up an EKS cluster and along with that, the cost for it. (this is just to showcase some terraforming + aws + architecture, so I try to keep costs low as I test it on a personal account)
2. Change the egress to S3 (for ALB logs and docker layers) and to ECS api to go via a VPC Endpoint so that we remove the need for a NAT Gateway in the VPC. Remember: Principle of least priviledge. The traffic to AWS services should not go through the internet but via a forced path by us.
3. I will move the certificate creation to the service terraform directory, so then the service can be ran many times independently of the main infrastructure bit.

Replacing ngrok with caddy + ssh

So I find ngrok.io to be an amazing service which I like a lot.

The downside is that I get a different sub-domain each time and that I don't have control over the whole thing.

I have googled for alternatives to ngrok and eventually I ended in this reddit and sequentially to this blog post from Jacob Errington.

This would have covered my needs, but I can't be bothered with setting up Letsencrypt and combining with nginx and stuff when I know of Caddy!

Caddy provides a reverse proxy with automatic LetsEncrypt certificates without the need to care more than writing a sime Caddyfile like the following:

sub.example.com         # the domain to be served
proxy / localhost:3333  # directive to proxy, and the target for proxying

And that's it!

Now you only need to save that file and make sure you set up caddy as a systemd service and load that file.Usually you will be getting 502's if you are not connected, or you can make sure to start it when you ssh into the box.

I will probably add more information on how to set up caddy to be a systemd service that auto starts and restarts on failure.

Hello Website

This is an example blog post. Not much here but that's not the point :)

lektor testing

This is a first attempt to use lektor for my blog. I used jekyll last time on github, lets see how this goes!

GPG signed commits on mac

To set up signed commits on mac:

• gpg --gen-key to generate the key

• gpg --list-secret-keys --keyid-format LONG to list your key plus its long key (after the slash after the 2048 or 4096 bit length)

• gpg --armor --export <PASTE_LONG_KEY_HERE> |pbcopy to copy to public key your clipboard so you can paste it in your profile.

• git config --global user.signingkey <PASTE_LONG_KEY_HERE> and git config --global commit.gpgsign true to add the key to your git config so you will be signing all your commits using that key. Make sure you want that setting to be global, or per git repo.

• add echo 'export GPG_TTY=$(tty)' to your .bash_profile or your .zshrc. this will make git ask you for your gpg passhprase every time you commit, to have it remembered, do the following:

• The following is specific for macs and will add the passphrase to your keychain so you won't be asked every time:

brew upgrade gnupg
brew link --overwrite gnupg
brew install pinentry-mac
echo "pinentry-program /usr/local/bin/pinentry-mac" >> ~/.gnupg/gpg-agent.conf
killall gpg-agent

Try echo "test" | gpg --clearsign for it to ask for your passphrase so it can be added to keychain. In the popup window make sure you tick the box to add to keychain.

Information collected from:

• github docs for signing commits
• github docs, adding key to git
• Stack overflow for mac specific gpg + passphrase