Split-horizon DNS and a wildcard cert - real TLS for internal-only homelab services

Preface

I run a handful of admin panels at home: TrueNAS itself, Nginx Proxy Manager, Pi-hole, Jellyfin, Paperless-ngx, Scrutiny and wg-easy.
None of them need to be reachable from the internet, and (now) none of them are.

However, they still all get a real, browser-trusted *.knng.de certificate instead of a self-signed one.

Here’s how it’s wired up, and how to set it up yourself if you want the same thing.

TrueNAS

Everything runs on a TrueNAS Community Edition server (192.168.1.15) with two pools:

• storage: a 2-wide mirror, 9.1 TiB usable, holds media and documents
• apps: a 2-wide mirror, 238 GiB, dedicated to app data

Apps run as TrueNAS’s built-in container workloads:

All six went in straight from the TrueNAS app catalog, no manual container setup needed (unless you’d rather not use ixVolumes, in which case that’s on you :D).

You’ll need to move the TrueNAS UI itself off the default 80/443, so those ports are freed for the reverse proxy.
Mine is on 8080 (HTTP) and 30000 (HTTPS).
You can do that under: System -> General Settings.

No public IPv4

My internet connection is Vodafone fiber with DS-Lite.
IPv4 is tunneled through IPv6 to the provider and NATed there with other customers.

Incoming IPv4 connections never reach my router, so there’s no port to forward.
But one service does need to be reachable from the outside: WireGuard.

Luckily, DS-Lite only affects IPv4.
Vodafone hands IPv6 directly to my router, without a tunnel and without sharing addresses.

The catch: my IPv6 prefix isn’t guaranteed to stay the same, so I need a DNS name that follows it.

The FRITZ!Box offers exactly that with its MyFRITZ! domain, and it would work fine.
But it ties the name to AVM, and a script with a domain I control is more “homelab” anyway. :D

The details, and how the MyFRITZ! way would work, are in the WireGuard part.

So here’s what runs instead:
A script on the NAS compares its current global IPv6 address with the one from its last run, a TrueNAS cron job runs it every X minutes.
It only calls the Cloudflare API when the address changed, so most runs do nothing.

If you want to do the same: You’d need to create a file called .cf_token that simply holds your Cloudflare API token.
See both files below and adjust paths & domains accordingly.

cfut_ABCDEF12gh34IJk5LMNO...

#!/usr/bin/env bash
set -euo pipefail

ZONE="<domain>.de"
RECORD="<subdomain>.<domain>.de"
TOKEN_FILE="/mnt/storage/scripts/.cf_token"
STATE="/mnt/storage/scripts/.cf_ddns_last"
LOG="/mnt/storage/scripts/cf-ddns.log"
API="https://api.cloudflare.com/client/v4"

log() { printf '%s %s\n' "$(date -Is)" "$*" >> "$LOG"; }

TOKEN=$(tr -d '[:space:]' < "$TOKEN_FILE")
[ -n "$TOKEN" ] || { log "ERROR empty token"; exit 1; }

# token stays off the process list
cfcurl() { curl -sS -K- "$@" <<EOF
header = "Authorization: Bearer $TOKEN"
header = "Content-Type: application/json"
EOF
}

# first non-ULA global address on any interface
IP=$(ip -6 addr show scope global | awk '/inet6/ && $2 !~ /^f[cd]/ {print $2}' | cut -d/ -f1 | head -1)
[ -n "$IP" ] || { log "ERROR no global IPv6"; exit 1; }

if [ -f "$STATE" ] && [ "$(cat "$STATE")" = "$IP" ]; then exit 0; fi

zid=$(cfcurl "$API/zones?name=$ZONE" | jq -r '.result[0].id // empty')
[ -n "$zid" ] || { log "ERROR zone lookup failed"; exit 1; }

rid=$(cfcurl "$API/zones/$zid/dns_records?type=AAAA&name=$RECORD" | jq -r '.result[0].id // empty')
[ -n "$rid" ] || { log "ERROR record lookup failed"; exit 1; }

resp=$(cfcurl -X PATCH "$API/zones/$zid/dns_records/$rid" --data "{\"content\":\"$IP\"}")
if [ "$(printf '%s' "$resp" | jq -r '.success')" = "true" ]; then
  printf '%s' "$IP" > "$STATE"
  log "updated $RECORD -> $IP"
else
  log "ERROR patch failed: $(printf '%s' "$resp" | jq -c '.errors')"
  exit 1
fi

This is the only record in the public zone that points at my home network.

DNS-01 through Cloudflare

Again: None of these hosts are reachable from outside, which rules out an HTTP-01 challenge, since that requires the ACME server to actually reach the host over the internet to validate it.

NPM instead requests the wildcard cert through a DNS-01 challenge, using a Cloudflare API token to create the temporary _acme-challenge TXT record on the knng.de zone. Let’s Encrypt validates against that record instead of hitting the server directly, so the cert gets issued without opening anything to the internet at all.

This is a one-time setup, done once and reused everywhere, under Certificates -> Add Certificate -> Let’s Encrypt via DNS, domain *.knng.de, provider Cloudflare:

Worth knowing before you paste anything in: that text box is a certbot credentials file, not a validated UI field.
Whatever you put in gets written to disk as-is and handed straight to the dns-cloudflare certbot plugin.

Because of that I scoped the token to Zone:DNS:Edit on just the knng.de zone rather than handing it anything broader.

The payoff is one *.knng.de certificate sitting in NPM’s certificate store, ready to be reused by every internal proxy host without going through any of this again.

Nginx Proxy Manager

NPM runs as a TrueNAS app and terminates TLS for every internal hostname:

Every host uses the same wildcard certificate for *.knng.de, with force-SSL, HTTP/2, and “block common exploits” all switched on.

Adding a new host is really just two form fields. Hosts -> Add Proxy Host, a domain name, the backend IP and port:

Then the SSL tab, where the existing *.knng.de wildcard just shows up as a certificate you already have. No new challenge, no new DNS record, just pick it and move on:

How a request actually gets routed

This part tripped me up when I first set it up.
If every request goes to Pi-hole, and Pi-hole simply points every internal name at the NAS, how the fuck does NPM know which service I actually want?

The answer is that two separate things happen, and DNS is only the first one.

1. DNS gets you to the NAS. I request app.knng.de and Pi-hole resolves it to 192.168.1.15. That’s all an A/AAAA record does: it maps a name to an IP and knows nothing about ports or services.
2. TLS gets you to the right service. My browser connects to that IP on port 443 and starts a TLS handshake, and the hostname I want travels along inside it.

Or, in more detail: NPM listens on 443 for every proxy host at once and redirects all plain HTTP to HTTPS, which is exactly why I moved TrueNAS itself off 443 in the first place.
That also means every *.knng.de subdomain arrives at the same IP and the same port, so neither of them can tell NPM which service I’m after.
The hostname in the handshake can, and that’s SNI, Server Name Indication, which I had missed. :D

It’s sent in cleartext as part of the TLS ClientHello, before any certificate or encrypted data changes hands.
NPM reads that value, matches it against its proxy host configs, and forwards to the right backend IP:port, the media server’s 30030, say.
Afterwards it checks the HTTP Host header of the actual request, which normally matches the SNI value anyway.

So DNS picks the NAS and SNI picks the service, which is also exactly why split-horizon DNS works at all here: it only ever has to control the first step.

The DNS split: public zone vs. local resolver

This is the part that actually ties everything together. The public Cloudflare zone for knng.de only contains what genuinely needs to be public:

• the blog itself (knng.de, a Worker route, since this site is built with Eleventy and deployed on Cloudflare Workers)
• a VPN subdomain, an AAAA record pointing at the WireGuard endpoint (IPv6-only, since DS-Lite means it has to be reached over IPv6 directly)
• mail records (MX, DKIM, DMARC, SPF)
• a site verification TXT record

None of the internal hostnames exist in public DNS at all. They only live in Pi-hole’s local DNS records, all pointing at 192.168.1.15, resolvable only for devices on my own network.

Under Settings -> Local DNS Records it’s just two plain text fields, a domain and an IP:

Pi-hole, a nice side effect

Pi-hole obviously still does what it says on the tin too, DNS-level ad and tracker blocking, with Cloudflare (1.1.1.1, DNSSEC) as upstream, running the StevenBlack hosts list plus HaGeZi’s Threat Intelligence and Multi Pro++ lists, as well as OISD’s Big List. That’s incidental to everything above, but it’s the same container quietly doing double duty as the local resolver for the whole setup.

Why bother with real certs for internal-only stuff?

I could just click through the self-signed cert warning every time, but the wildcard-cert-via-DNS-challenge approach costs nothing extra to run once it’s set up, and every internal tool just works.

It also fixed something that annoyed me for a long time: Apple’s Passwords app couldn’t tell my services apart when I opened them as http://IP:Port.
Logging in to TrueNAS at 192.168.1.15:30000, it would happily suggest the password for another service on the same NAS (192.168.1.15:<other port>).
Now that every service has its own hostname (truenas.knng.de, pihole.knng.de, …), each one gets its own entry and the right password.

But most importantly: it simply seemed interesting to me :D
It also generalizes going forward. Any new *.knng.de subdomain gets a valid cert automatically, whether or not it’s ever meant to be public.

Cheers.