Y3K LAB/GUIDES

Build Resilient Home DNS with Pi-hole

Last verified July 17, 2026View as Markdown

Build Resilient Home DNS with Pi-hole

Pi-hole is the single most outage-sensitive service in most homelabs — when it goes down, every device loses name resolution at once, and it looks like the whole internet is down. This is a defense-in-depth pattern that makes it genuinely resilient: two Pi-holes behind a floating VIP, plus the parts most failover guides skip. Battle-tested in a real homelab.

Why this exists

Pi-hole is the worst single point of failure in a homelab. When it dies, every device on the network loses name resolution — no streaming, no smart home, no browsing — and the failure looks like “the whole internet is down” to everyone in the house. It’s the one service where an outage is immediately, universally noticed, usually by someone who can’t fix it.

The community’s answer is well-established: two Pi-holes and a floating VIP managed by keepalived (VRRP). That part is not new — there are many good guides (see Prior art). This writeup assumes you can follow one of those for the base VIP, and focuses on the five things those guides almost universally skip — the parts that turn a VIP demo into something you’d actually trust your household to:

  1. Hand out only the VIP over DHCP (and why a “backup” DNS server makes things worse).
  2. A health check that can’t false-pass.
  3. Local self-heal so a crashed daemon comes back on its own.
  4. An independent monitor whose alert path doesn’t depend on DNS.
  5. Testing the failure, not the happy path.

Together they form a layered system where each layer catches what the others can’t.

The base pattern (one paragraph)

Two Pi-holes (192.168.1.51 and 192.168.1.52). A third, floating VIP 192.168.1.53 (mnemonic: 53 is the DNS port). keepalived runs on both; the higher-priority node (.51, MASTER) holds the VIP, and if it fails the VIP moves to .52 (BACKUP) in ~3 s. Clients are pointed at the VIP and never talk to a real Pi-hole IP directly. That’s the foundation. Everything below is what makes it hold up in the real world.

   Clients ──DNS──▶ VIP 192.168.1.53      (DHCP hands out only .53)
                         │
                         │   keepalived VRRP keeps the VIP on the healthy, higher-
                         │   priority node; on failure it floats to the other in ~3 s.
      ┌──────────────────┴──────────────────┐
      ▼                                     ▼
      Pi-hole #1 192.168.1.51               Pi-hole #2 192.168.1.52
      MASTER, prio 150                      BACKUP, prio 100
      self-heal: keepalived                 self-heal: keepalived
        Restart=on-failure                    Restart=on-failure

   Independent watcher — a third box that hosts NEITHER Pi-hole:
     every 60 s  dig @192.168.1.53  ──(debounced)──▶  alert on a DNS-free path

1. Hand out only the VIP — drop the “backup” DNS server

The instinct — and what most guides show — is to hand out two DNS servers via DHCP: dns1 = VIP, dns2 = one of the real Pi-holes. It feels like belt-and-suspenders. It quietly defeats the entire point of the VIP.

DNS clients do not reliably treat dns2 as “only use if dns1 times out.” Many resolvers (glibc with rotate, systemd-resolved, plenty of IoT stacks) round-robin or spray across the whole list. So a meaningful fraction of your devices end up talking to a real Pi-hole IP directly — bypassing the VIP. When that box dies, those clients get the exact per-query timeout stall the VIP was built to eliminate, and they get it for the entire outage (a directly-named dead resolver stays dead), not just the ~3–6 s it takes the VIP to move.

We measured the stall on the naive two-server setup: ~45× slower (5 fresh lookups = 15,381 ms with a dead dns1 vs 341 ms healthy). The fix is counter-intuitive but correct: hand out the VIP and nothing else. The VIP is the redundancy; a second address just gives clients a way to route around it.

If you’re nervous about “no fallback,” that anxiety is real but it points at a different gap — see layer 4 (the monitor) and self-heal. The answer is to make the VIP reliable, not to give clients a way to bypass it.

2. A health check that can’t false-pass

keepalived should move the VIP based on “is this box actually answering DNS?”, not “is the box powered on?” That means a vrrp_script that queries the local resolver and only succeeds on a real answer. Everyone knows this much.

The trap is in the check itself. The obvious version —

out=$(dig +short pi.hole @127.0.0.1 2>/dev/null); [ -n "$out" ]   # WRONG

— reports healthy while the resolver is dead. When port 53 refuses the connection, dig +short prints ;; communications error … connection refused to stdout, not stderr, so 2>/dev/null doesn’t suppress it and the output is non-empty. The check “passes” on a corpse, the VIP never moves, and DNS stays black.

The fix is to validate that the answer is an IPv4 address, which the error text can’t satisfy:

dig +short +time=1 +tries=1 pi.hole @127.0.0.1 2>/dev/null | grep -qE '^[0-9]{1,3}(\.[0-9]{1,3}){3}$'

We only caught this because we tested the failure (see layer 5) — a naive check “passes” in the happy path forever.

3. Local self-heal

keepalived’s packaged systemd unit ships Restart=no. So if the keepalived process crashes, systemd leaves it dead — that box now has no VRRP, and if it happened on both boxes the VIP would go unheld and DNS would go dark, with nothing trying to recover. A one-line drop-in fixes it:

# /etc/systemd/system/keepalived.service.d/override.conf   (on BOTH boxes)
[Service]
Restart=on-failure
RestartSec=5

Now a crashed keepalived self-heals locally — no network, no coordinator, no waiting on a human. (Check your Pi-hole’s pihole-FTL unit too; on current releases it already ships Restart=on-failure.) systemd’s default start-limit (5 starts / 10 s) still halts a genuine crash-loop — e.g. a bad config a restart can’t fix — which is correct: the loop stops and the monitor pages you.

Note there’s nothing to “un-heal”: a restarted daemon just re-reads its normal config, and the VIP returns to the master automatically via VRRP preemption (the higher-priority node reclaims it on recovery). The health-check priority penalty un-applies itself. Recovery is entirely hands-off.

4. An independent monitor whose alert path doesn’t depend on DNS

VRRP failover handles a single box dying. Self-heal handles a crash. Neither catches the tail cases — a config error that kills keepalived on both boxes, both nodes down, or a resolver that’s “up but wrong.” For those you want to be told. Three design rules make the monitor trustworthy, and all three are routinely gotten wrong:

Our implementation: a 60-second systemd timer on the cluster’s third node runs the dig @VIP check, debounces (alert only after 3 consecutive failures ≈ 3 min, so a healthy ~6 s VIP failover never pages), and on sustained failure POSTs to a Home Assistant webhook at a raw IP (http://<HA-IP>:8123/api/webhook/…). HA then pushes to a phone, drops an in-app notification, and logs a breadcrumb; a recovery event clears it. The webhook is a low-power trigger token (its only capability is to fire that one notification) rather than a full API credential on the watcher — least privilege.

5. Test the failure, not the happy path

A health check or a self-heal that has only ever been observed working is untested. A check that passes when things are fine tells you nothing; you have to see it fail on the actual fault. So: kill the resolver and confirm the VIP moves. Kill keepalived and confirm systemd restarts it. Point the monitor at a dead IP and confirm it actually alerts (and that the alert arrives). Prove the broken state first, then prove the fix — otherwise you’ve built a smoke detector you’ve never held a match to. (This is how we found the stdout-false-pass bug in §2 and confirmed the debounce in §4.)

The layered result

Defense in depth: each layer catches a different failure class, and the ones a layer can’t fix fall through to the next.

Failure Caught by Outcome
One Pi-hole box powers off / reboots VRRP failover VIP moves to survivor in ~3 s; most clients never notice (DNS cache)
A resolver is up but not answering (FTL dead, box on) Health-checked VRRP Priority drops, VIP moves to a box that actually resolves
keepalived (or FTL) process crashes Local self-heal (Restart=on-failure) Daemon back in ~5 s, locally, no human
Bad config kills keepalived on both boxes; both nodes down; “up but wrong” Independent monitor You get paged (self-heal correctly gives up on a crash-loop)
The alert itself during a DNS outage Raw-IP path + local channel Alert still fires because it never needed DNS

The failure that has no automatic fix — a bad config pushed to both boxes — is the one the monitor exists for, and it’s the one a human should handle anyway (the fix is “revert,” not “restart”). Everything above it recovers on its own.

Hardening roadmap (where to take it next)

Honest about the current build’s edges, roughly easiest-first:

Gotchas, condensed

Prior art & further reading

The keepalived + Pi-hole VIP base pattern (often paired with gravity-sync for list replication) is well documented; credit where due:

What this writeup adds to that body of work: VIP-only DHCP (and why a backup server hurts), a health check that can’t false-pass, local self-heal, and an independent monitor whose alert path survives the very outage it reports — assembled as one layered system, with the reasoning and the failure-first testing that most guides skip.

How it’s deployed

This runs on a 3-node cluster (Proxmox, in our build): the two Pi-holes are LXCs pinned to different nodes, so losing one node can’t take out both, and the monitor runs on the third node — sharing no fate with either resolver. Keep the keepalived configs, the self-heal drop-in, and the monitor’s timer/script in version control; the failure-first tests (§5) are what prove the whole thing actually recovers.

← All guides

Host a Free Website with Cloudflare Pages and GitHub

Read the guide

Restrict a Website to Specific People with Cloudflare Access

Read the guide

Set Up Split-Horizon DNS at Home

Read the guide