When Our Leak Test Invented a Leak.

What it looked like

An Android phone on 5G. Private DNS set to our hostname and nothing else. No browser-level secure DNS, no VPN. We ran our own DNS leak test and it reported four resolvers:

162.159.96.6     Cloudflare
162.159.96.7     Cloudflare
192.178.94.16    Google LLC
172.253.94.153   Google LLC

Cloudflare is one of our two configured upstreams, so those two were expected. Google was not. We removed Google from our resolver configuration a long time ago on privacy grounds - its entry is still sitting commented out in our config. There is no code path in our stack that sends a query to Google.

Two more facts made it stranger rather than clearer:

• • Running two third-party leak tests from the same phone, at the same time, showed no Google at all.
• • Running our own test again with a local packet-level DNS interceptor active produced a clean result.

So the leak appeared only in our own test, only sometimes, and only on the phone.

Why Google made no sense

Our resolution chain is short and we can account for every hop:

phone → DoT → AdGuardHome → unbound → dnscrypt-proxy → { Cloudflare, Quad9 }

AdGuardHome has exactly one upstream - unbound - and an empty fallback list, so there is no escape hatch. unbound forwards everything to dnscrypt-proxy. And dnscrypt-proxy is pinned to two upstreams by static stamp.

We pushed nine correctly-formed probe queries through that exact chain and captured what reached the authoritative side: Cloudflare only. No Google.

So the chain was clean. The Google entries were arriving by some other route.

The red herring worth naming

Our parent zone is authoritative-hosted on Google Cloud DNS. The nameservers are literally ns-cloud-e1..e4.googledomains.com.

It is very tempting to connect that to Google IPs appearing in a DNS capture, and it is wrong. An authoritative nameserver answers queries; it does not go out and query your servers. Google Cloud DNS hands a recursor the delegation and then plays no further part. Two unrelated things called “Google” in the same system is a coincidence, and we wasted time on it so you do not have to.

The real identification was simpler. We resolved a probe deliberately through 8.8.8.8 and captured what arrived at our nameserver:

192.178.93.155   192.178.92.20   172.253.94.152
192.178.93.149   192.178.92.30   172.253.94.153   ← reported verbatim

172.253.94.153 is one of the two IPs the phone reported, and 192.178.94.16 sits in the same block. These are Google Public DNS recursive egress addresses. Something, somewhere, had asked Google to resolve our probe.

The 57 milliseconds that explained it

Our capture pipeline stores only session → set of IPs; it discards the probe index. To find out which probe leaked we had to capture packets directly.

For a single probe, in one session:

20:45:20.556 → .750   Quad9 queries our nameserver   ×18 attempts in 194ms
20:45:20.807          Google 172.253.94.152           ← 57ms after Quad9 stopped
20:45:20.820          Google 192.178.92.21

Quad9 did not leak anything to Google. Quad9 asked our authoritative nameserver eighteen times in under two hundred milliseconds, got nothing usable, and gave up. Fifty-seven milliseconds later the phone retried the same name against a secondary resolver - on that carrier, Google Public DNS. Google recursed, hit our nameserver, and our capture faithfully recorded Google's egress address as a resolver that had queried us.

Which it had. It just was not a leak.

The actual mechanism

Our probe zone is delegated to our own host. But port 53 on that host runs a recursive resolver with no authority for the zone.

So when a probe query arrived, it did what a resolver does: it forwarded the query upstream. The upstream recursor looked up the delegation, found our host, and came straight back to us. We forwarded it again.

recursor → our :53 → forwarded upstream → recursor → our :53 → ...

A resolution loop. It always ended in SERVFAIL.

dig probe.<zone> @<our-ns>   → status: SERVFAIL
dig SOA <zone>  @<our-ns>    → status: SERVFAIL

Nothing was ever authoritative for a zone that was delegated to us. That single fact produced every symptom:

• • The retry storm. Recursors retry hard against a nameserver that will not answer.
• • The phantom resolver. SERVFAIL reaches the client, the client falls back to a secondary resolver, and the capture records it as a leak.
• • Why only some probes. Only the ones whose failure landed inside the fallback window.
• • Why the packet-level interceptor was clean. It resolves queries itself and returns failures straight to the app; the OS multi-resolver fallback never engages, so there is no second path to record.
• • Why other leak tests showed nothing. Their authoritative servers answer.

None of this stopped the test from working. It kept finding real leaks and reporting real resolvers throughout - that is why the defect survived. What it added, intermittently, was one resolver too many: the instrument occasionally manufacturing the very fault it was built to detect, on top of results that were otherwise correct.

Why every obvious fix destroys the test

This is the part worth your attention, because the obvious fixes all look correct and all pass a casual test.

Three of them, in increasing order of sophistication:

• 1. A rewrite in the recursive resolver, mapping the probe zone to an address.
• 2. A per-domain upstream sending the probe zone to a small local responder.
• 3. A local-zone or stub-zone in the recursive layer.

Every one of them terminates the loop. Every one of them returns NOERROR instantly. And every one of them silently destroys the leak test for your own users.

Here is why. A leak test only works because the user's probe leaves your building, reaches a third-party recursor, and that recursor then queries your authoritative nameserver - which is the moment you capture its address. All three fixes answer the probe inside your own users' resolution path. The probe never egresses, no third-party recursor is ever involved, and there is nothing to capture. Your own users get an empty result.

Users on encrypted transports fare worst: their queries never touch the plain DNS port the capture watches, so they see literally nothing.

We did not reason our way to this. We built an isolated resolver instance with the local-zone fix applied and measured it:

probe query                    → NOERROR, answered instantly
packets sent to the forwarder  → 0

Zero. The probe never left.

The hypothesis we discarded

Our first real proposal was to split on the RD (recursion desired) bit. The theory is clean: stub resolvers set RD=1 because they are asking someone to do work for them; a recursor querying an authoritative server sets RD=0 because it only wants that server's own data. Answer RD=0 authoritatively, forward RD=1 normally, everyone is happy.

We tested it against real captured traffic before building anything:

The stub half held perfectly - 68 out of 68. Cloudflare was textbook. Quad9 sends RD=1 on roughly half its queries, which breaks the theory: those queries are indistinguishable from a stub client's by that test alone.

We kept the RD test, because it is what protects ordinary clients, and added a second condition rather than pretending the first was sufficient.

What we actually did

The two directions are only distinguishable at the packet layer:

• • a query arriving inbound on the DNS port is a recursor asking us as the authoritative nameserver
• • a query leaving outbound through our encrypted forwarder is our own user's probe on its way out

So we redirect only the inbound case to a small authoritative responder, and leave outbound recursion completely untouched. Client probes still egress, still reach Quad9 or Cloudflare, and are still captured. Recursors get an immediate authoritative answer and stop retrying.

The redirect fires when the query is for the probe zone and either:

• • the RD bit is clear (a recursor-style query), or
• • the source is within our own upstreams' egress ranges - because a probe-zone query arriving from one of our own upstreams is by definition a recursor asking us as the authoritative server, never one of our own clients.

The responder itself is deliberately boring: it answers A for any name in the zone, returns proper NODATA with an SOA for everything else, sets the authoritative answer bit, and refuses anything outside the zone.

That NODATA detail matters more than it looks. Browsers issue HTTPS resource record queries for ECH and HTTP/3 discovery - we watched Cloudflare send one per probe on behalf of one desktop browser. An unanswered HTTPS query retries exactly like an unanswered A query. Answering only A would have left half the storm in place.

Two traps in the implementation

Both cost us time and neither is obvious.

The capture must run before the redirect. Packet capture and address translation happen at different stages of the firewall. If the redirect rewrites the destination port first, the capture rule stops matching and your leak test goes quiet - with no error anywhere. Ours was already at the earlier stage by luck, not design. Check the ordering explicitly rather than assuming it.

Untracked packets skip translation entirely. High-traffic DNS servers commonly mark port 53 traffic as untracked to keep connection-tracking tables small. Address translation needs that tracking. The narrow exemption has to be ordered ahead of the rule that disables tracking, or the redirect silently never fires.

We also lost twenty minutes to a self-inflicted one: our capture matches the probe name at a fixed byte offset, so it only matches when the probe label is exactly the expected length. Our hand-typed test probes were one character too long, matched nothing, and made a working fix look broken. The rule counters - not the logs - gave it away.

Results

Measured across real devices, before and after:

Verified on Android over 5G on two carriers, and on desktop in two browsers. The 33-query figure is worth dwelling on: that was after our first fix, which had already eliminated the user-visible symptom. The load was still there, invisible, until we measured again. Fixing the symptom is not the same as fixing the cause.

If you tested with us before 28 July 2026

The general lesson

There is a specific lesson and a general one.

The specific one: if you run a DNS leak test, check that your authoritative nameserver actually answers. A zone delegated to a host running only a recursive resolver will SERVFAIL, and a SERVFAIL leak test does not fail closed - it fails by inventing plausible, wrong results. This is a property of the technique rather than of any particular implementation, and it is worth checking wherever the technique is used.

The general one is older and keeps being true: a measuring instrument that perturbs what it measures will hand you confident, well-formatted, wrong answers. Ours had been doing exactly that, and the only reason we caught it is that the wrong answer happened to be one we knew was impossible. Had it reported a plausible resolver, we would have believed it.