Traefik behind Traefik

Having a server at home means sharing the personal IP address with the world. This is not always what you want, because certain databases like maxmind.com are pretty accurate in mapping IP addresses to locations.

This issue first came to my mind the moment I discovered the browser extension Flagfox which displays a little flag in the browsers search bar to give a hint on the location of the websites server.

At first, I was using Cloudflare to hide the IP address of my second server, but it turns out that Cloudflare and Deutsche Telekom have a very bad peering which leads to the problem that my services behind Cloudflare were extremely slow. This is kinda funny, because I initially thought, Cloudflare could enhance my page speeds with their caching mechanisms.

Using my Hetzner VPS as a proxy

The IP address of my Hetzner VPS is static, the location is always Nuremberg and I do not have a problem with everyone knowing that. Thus, I decided to use my VPS as a proxy for my second server at home.

Traefik is my reverse-proxy of choice for both my servers and thanks to the proxy protocol, it is possible to have two Traefik instances chained behind each other while preserving certain information like the original IP address of the client.

Server 1 represents the proxy server on the Hetzner VPS, while Server 2 is my private server at home.

Here are the things that need to be configured for this setup:

DNS

Change the DNS records that originally pointed directly to Server 2 to point at Server 1 instead.

Server 1 (VPS)

Create a dynamic configuration file:

# proxy.yml on server 1
tcp:
  routers:
    router1:
      rule: >
        HostSNIRegexp(
        `sub.example.com`,
        `sub2.example.com`
        )
      service: service1
      tls:
        passthrough: true
      entrypoints:
        - websecure

  services:
    service1:
      loadBalancer:
        servers:
          - address: "{{ getHostByName `dyndns.example.com` }}:443"
        proxyProtocol:
          version: 2

As all traffic now first passes Server 1, you need to declare those FQDNs that should be forwarded to Server 2 in the HostSNIRegexp. TLS should initially be terminated on Server 2, because we do not want to send unencrypted traffic over the internet. This is why TLS-passthrough should be enabled.

Traefik TCP loadbalancers always need the direct IP address where they should send the traffic to. They do not perform domain name resolution per default. This is a problem, because the IP address of Server 2 is not static and changes from time to time. We could write it down there and it would work initially, but after some time, the IP address changes and the traffic is routed into the void or at another server. Luckily, Traefik supports Go Templating in dynamic configuration files. It turns out that there is even a sprig function called getHostByName, which does exactly what we want and replaces a given FQDN with the corresponding IP address. Using Dynamic DNS, it can be ensured that the current IP address of Server 2 is always known.

But there is still a problem: The sprig function is only evaluated when the file is re-read by Traefik. This is the case when Traefik is restarted or the file is changed. With a simple crontab, we can make Traefik execute the function regularly and update the IP address.

# crontab -e
* * * * * touch /path/to/traefik-config/proxy.yml

Server 2 (Homelab)

Edit the entrypoint in the static configuration file (e.g. `traefik.yml`) and add Server 1 as trusted IP for the proxy protocol.

# traefik.yml on server 2
entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
  websecure:
    address: ":443"
    proxyProtocol:
      trustedIPs:
        - "<static IP of server 1>"

Result

In the end, it looks like services, which are hosted in my home on Server 2 (like this blog), live on Server 1 in Nuremberg instead. The Flagfox and Geotool informations are no longer correct 🎉

Alternatives

Using a wireguard tunnel to bring the traffic to the second server might also have been an option, but I wanted to keep it simple and did not want to introduce another service.