How I access the services I self-host in my cluster

For the most part, all of the services that I self host in my homelab cluster are all deployed on their respective subdomains on my domain khzaw.dev.

jellyfin.khzaw.dev
grafana.khzaw.dev
vaultwarden.khzaw.dev
uptime.khzaw.dev

Barring a few exceptions, all of these DNS would resolve to an internal private IP, 10.0.0.231. Naturally, they are not accessible outside of my home LAN.

What is this 10.0.0.231 address?

This 10.0.0.231 address is the cluster’s main entry point. It belongs to the ingress-nginx-controller service. The ingress controller is exposed as Kubernetes Service of type LoadBalancer.

spec:
  type: LoadBalancer
  loadBalancerIP: ${INGRESS_VIP}

In my cluster settings, INGRESS_VIP is 10.0.0.231. In a cloud provider environment, a LoadBalancer Service usually asks the cloud provider to provision a real load balancer and attach an external IP to it. In a bare-metal homelab, nobody does this. This is what MetalLB does. I have it configured with a small pool from my LAN subnet.

addresses:
  - 10.0.0.230-10.0.0.254

Since 10.0.0.231 is inside that pool, MetalLB can use it for ingress-nginx-controller Service. It updates the Service’s load balancer status with that IP, and then advertises the IP on the LAN using Layer 2 mode.

Layer 2 mode?

In Layer 2 mode, MetalLB picks one node to answer for a given service IP. For 10.0.0.231, the speaker on that node responds to ARP requests on the LAN. So the rough sequence is

• A client watns to reach 10.0.0.231
• It broadcasts ARP: “Who has 10.0.0.231?”
• The MetalLB speaker on the selecte dnode replies with that node’s MAC address.
• The client sends packets for 10.0.0.231 to that node.
• Kubernetes Service routing then forwards the traffic to the ingress-nginx-controller pod.

Now the packets arrive at the ingress-nginx-controller pod. This is where the hostname-based routing happens. When a request arrives, nginx looks at the HTTP Host header and send the traffic to the right Service.

At home versus away from home

In my LAN network, there is not much to it. My device asks for say jellyfin.khzaw.dev, gets 10.0.0.231, and sends traffic over the LAN. When I am away from home, the DNS answer is the same, so it wouldn’t work. This is where Tailscale comes in. I use a Tailscale subnet router for that. The cluster advertises a few specific home-network routes, including the ingress IP.

apiVersion: tailscale.com/v1alpha1
kind: Connector
metadata:
  name: homelab-subnet-router
spec:
  exitNode: true
  hostname: homelab-subnet-router
  subnetRouter:
    advertiseRoutes:
      - ${INGRESS_VIP}/32
      - ${NAS_IP}/32
      - ${ROUTER_IP}/32

And I installed Tailscale on all my devices. So, when jellyfin.khzaw.dev resolves to 10.0.0.231, Tailscale knows how to get packets for 10.0.0.231 back to my house.

At home:
  app.khzaw.dev -> 10.0.0.231 -> LAN -> ingress-nginx

Away from home:
  app.khzaw.dev -> 10.0.0.231 -> Tailscale -> ingress-nginx

In each devices, you can configure Tailscale to auto turn on when it is outside a certain WiFi SSID. The whole setup is pretty frictionless.

As you can see, I also advertises my Router IP and NAS IP. So that, for whatever I need to access them, I can access them away from home. I also activated exitNode in the off chance that I am visiting a place that has internet restrictions, I can use my homelab cluster as my VPN.

Public things are separate

Most of the self hosted services in the cluster are private. For the few things that are public, like this blog, I use Cloudflare Tunnel. cloudflared runs in the cluster and connects out to Cloudflare. The blog route looks like this.

- hostname: blog.${BASE_DOMAIN}
  service: http://blog.default.svc.cluster.local:8080

So public traffic comes through Cloudflare Tunnel and goes straight to the in-cluster Service I configured.

Where it settled

This is where it has settled for now. I am quite happy with the way things are right now. Most days I type the normal URL and forget whether I am home or on Tailscale.