When UDP Clamping Exposes a TCP Bug.

Where part one ended

The short version, for anyone arriving here first: on 12 July a spoofed flood used our public resolver to reflect 2,257-byte answers at victims in Brazil, 31.8× more traffic than the attacker spent. We built size-gated response rate limiting with a TC=1 slip into our AdGuard Home fork, and the reflection collapsed - 98.2% of it. The full story is here, and this article assumes you know how the RRL works.

Part one also ended with two open items on its own defence list: clamp the EDNS UDP payload size to 1232, and consider DNS Cookies. And it carried one honest caveat: the RRL still answers 5 queries per second per bucket, and those answers were still full-size - 31.8× amplification on that answered slice, for as long as a flood runs.

Deploying those two items is what this article is about. The clamp took one constant. Verifying it took three days, because the verification kept finding things.

The helpful proxy: why our internal clamp did nothing

Here is the part that makes the clamp story worth telling: our backend was already compliant. Behind the public dnsproxy edge sits unbound, and unbound had been Flag-Day-clamped for ages - max-udp-size: 1232, edns-buffer-size: 1232, harden-short-bufsize: yes. By every hardening checklist, the box was done.

It made no difference whatsoever. Querying the public edge with dig +bufsize=10000 still returned full 2,257-byte answers, and the response OPT echoed udp: 10000 straight back.

The mechanism is almost funny. unbound does truncate at 1232 with TC=1 - toward dnsproxy, over loopback. And dnsproxy does exactly what a good proxy is supposed to do with a truncated answer: it retries over loopback TCP, receives the full answer, and then serves that answer out over public UDP, honoring whatever buffer size the client advertised. The internal clamp is invisible from outside. (The cache holds the full answer afterwards, but the cache is not the bug - turn it off and the bypass still works. This is a TCP-retry bypass, not a caching one.)

The lesson generalizes well past DNS: a security control on a hop the attacker cannot reach is a control you do not have. Audit the hop that faces the wire, not the config files behind it.

The fix: one constant, two places

The change lives in our public dnsproxy fork (branch edge-udp-pool, commit e1cef22), in proxy/dnscontext.go:

// DNS Flag Day 2020.  Amplification floods advertise huge buffer sizes
// (observed live: 10000) to reflect maximal answers at spoofed victims.
const maxAdvertisedUDPSize = 1232

// 1. dnsSize() - the size the response is actually truncated to:
size16 = min(o.UDPSize(), maxAdvertisedUDPSize)

// 2. calcFlagsAndSize() - the size echoed back in the response OPT, so we
//    never advertise more than we honor:
dctx.udpSize = min(o.UDPSize(), maxAdvertisedUDPSize)

Both places matter. Clamping only the truncation size while parroting udp: 10000 back at the client is a resolver lying about its own limits - the client keeps believing large UDP answers are possible and keeps asking for them.

Measured on the live edge, same query, raw UDP (dig cloudflare.com TXT +bufsize=10000 +ignore):

The residual amplification on the RRL's 5-per-second answered budget - the honest caveat of part one - drops from 31.8× to ~16.7×. Halved. TCP, DoT, DoQ and DoH are untouched: they are connection-verified and unspoofable, and still serve full-size answers.

TC=1 is a promise. We tested it, and it broke.

TC=1 is not a refusal. It is a promise: "come back over TCP and I'll give you the whole thing." The clamp makes that promise constantly, for every answer over 1,232 bytes, so verifying the clamp meant testing the promise with genuinely large answers. The test corpus, all measured live on 13 July 2026:

Every answer above 4 KB failed. The boundary was exactly 4 KB, and it failed over TCP too - so this was not the clamp. The clamp merely exposed it: until responses started truncating over UDP, nobody had ever needed the TCP recovery path for an answer bigger than 4 KB. The bug had been sitting there silently for as long as the chain existed.

Walking it hop by hop localized it fast: the public edge said SERVFAIL, unbound on :5353 said SERVFAIL, and dnscrypt-proxy on :5053 - the last hop before the encrypted upstreams - simply timed out.

The smoking gun

tcpdump on the upstream interface while asking for cisco.com TXT. Quad9, speaking DNSCrypt over TCP port 8443, delivers the complete encrypted answer. We acknowledge every byte. Then we reset the connection we asked for:

149.112.112.10.8443 > us: Flags [P.], seq 1:4345,    length 4344
149.112.112.10.8443 > us: Flags [.],  seq 4345:5793, length 1448
149.112.112.10.8443 > us: Flags [P.], seq 5793:6979, length 1186   ← 6,978 B, all of it
us > 149.112.112.10.8443: Flags [.],  ack 6979                     ← we ACK it all
us > 149.112.112.10.8443: Flags [R.], seq 323, ack 6979            ← then we RST

This is the classic signature of a reader with a buffer smaller than the message. The upstream did nothing wrong. We hung up on our own answer.

Root cause: one legacy constant

MaxDNSPacketSize = 4096   // used for queries, UDP buffers, padding AND responses

One constant governing everything: query sizing, UDP buffers, DNSCrypt padding, and - the problem - the TCP response path. The TCP frame reader sized its buffer from it and rejected any frame above it, even though DNS-over-TCP frames are legal up to 65,535 bytes (RFC 1035 §4.2.2). Upstream dnscrypt-proxy still ships this. Nobody notices, because A and AAAA answers are small - it only bites on big TXT, DNSKEY and HTTPS record sets, and then it looks like a flaky upstream rather than a local bug.

The fix, in our public dnscrypt-proxy fork (branch edge-stable, commit 5fc5283e): a separate constant, applied to the response path only - the TCP frame reader, the DNSCrypt decrypt size gate, and the three response-validation gates.

// TCP frames are legal to 65,535 B (RFC 1035 §4.2.2).  Applied to responses
// only: queries, UDP buffers, EDNS advertisements and DNSCrypt query padding
// deliberately keep 4096/1252 - raising those invites fragmentation and
// padding bloat.
MaxDNSTCPPacketSize = 65535

The frame reader itself was rewritten to read the 2-byte length prefix, check it against the new ceiling, and then read exactly that many bytes into an exactly-sized buffer:

var lenBuf [2]byte
if _, err := io.ReadFull(*conn, lenBuf[:]); err != nil {
        return nil, err
}
packetLength := int(binary.BigEndian.Uint16(lenBuf[:]))
if packetLength > MaxDNSTCPPacketSize { return nil, errors.New("Packet too large") }
if packetLength < MinDNSPacketSize    { return nil, errors.New("Packet too short") }

buf := make([]byte, packetLength)
if _, err := io.ReadFull(*conn, buf); err != nil {
        return nil, err
}
return buf, nil

To be precise about what this is: a correctness fix. The old code was not byte-by-byte slow, and the new code is not a performance rewrite - it removes the 4 KB ceiling and allocates exactly what each response needs, which for small responses happens to be less than the old fixed buffer. That side effect is worth this one sentence and no more.

After the fix, cisco.com TXT returns all 88 records - 6,816 bytes - at every hop, and the two changes compose exactly as designed: UDP answers truncate at 1,232 with TC=1, the TCP retry delivers the full answer. The promise TC=1 makes is now one the chain can keep.

DNS Cookies: a VIP pass, not a gate

The final layer is RFC 7873 DNS Cookies. Before building anything we audited what we already had, because most operators will assume they are covered. Ours looked covered:

• • unbound: answer-cookie: yes - and completely inert. unbound only ever sees loopback traffic from dnsproxy, so its cookies can never bind to a real client address. Our own config comment even said "currently 0 clients send valid cookies." Of course it did.
• • net.ipv4.tcp_syncookies = 1 - a different technology entirely (TCP handshake floods), constantly confused with DNS Cookies. Already on, irrelevant here.
• • nftables, XDP, nginx - wrong layer entirely.
• • The public UDP:53 edge - the only hop a spoofed packet can reach - had nothing.

Same lesson as the clamp, and it deserves to land twice: compliance in the wrong place is not defense.

The mechanism, briefly. The server mints a cookie bound by HMAC-SHA256 to the client's source address (the RFC 9018 layout: version, reserved, 32-bit timestamp, 8-byte truncated MAC over the client cookie, that header and the client IP). Only a client that actually receives our response at that address can echo the cookie back. A spoofer never sees the response, so it can never present a valid cookie. It is stateless - no per-client storage, one secret, microseconds per packet.

This also inverts the residual honesty of part one. The 5-per-second answered budget is shared between the flood and any real client asking the same name - cookies let legitimate heavy clients identify themselves out of that shared budget, leaving the flood to compete for it alone.

The bug we had to fix to make it work

An RRL slip strips the response to a bare TC=1 header - answer, authority and extra sections all nulled. But the extra section is where the OPT record lives, and the OPT is where the cookie lives. Slipping a first-contact client would destroy the very cookie it needs to learn - and that TC=1 answer is exactly the vehicle by which a real client learns its cookie and escapes the slip on retry (RFC 7873 §5.2.3).

So the slip now keeps the cookie and nothing else. cookieOnlyExtra() in our AdGuard Home fork strips padding, EDE, client-subnet, everything - and preserves only the OPT's COOKIE option. A cookie-less slip is 27 bytes with no OPT at all (against a 71-byte query: 0.38×). A first-contact slip is 66 bytes, carrying the freshly minted server cookie - still under 1×. The defense had to leave a door in the wall it built.

The live demonstration

We deliberately exhausted the rate-limit bucket for (cisco.com, TXT) - that qname only, so no real traffic was affected - and fired the same query from the same source in three client states:

The three rows are worth labelling precisely, because dig 9.18 sends a client cookie by default whenever EDNS is on - a plain dig is the middle row, not the top one, and that default is also why "nobody sends cookies" is folklore. The belief comes from measuring at the wrong hop; see our own inert unbound comment above. Counters at the moment of the test: valid=1, client_only=42, stale=0, invalid=0, bypassed_total=1.

The security property that makes the bypass safe: the same cookie replayed from a different source address fails HMAC validation, is counted invalid, is denied the bypass and stays slipped. To be plain about the evidence: this is proven in the test suite (TestCookieBypass_SpoofedSourceCannotUseAnotherIPsCookie), not on the live wire - our upstream provider blocks source spoofing, so we cannot fire a spoofed packet at our own edge to demo it.

Rollout followed the same discipline as the RRL in part one: AGH_DNS_COOKIE=1 mints, validates and counts only - we verified live that a valid cookie changed nothing while the counters moved - and only then did AGH_DNS_COOKIE_BYPASS=1 arm the single behavioural change. The secret is fixed in the environment file, so minted cookies survive a restart.

The AdGuard Home fork carrying this is private; the design is documented in the public AdGuardHome-edge spec repo, alongside the RRL design from part one.

What the verification bought

Nothing here defeated the botnet. We still cannot see the attacker, the queries can start arriving again any night, and part one's closing sentence still stands: the job was never to beat them, it was to refuse to be the weapon.

What this part adds is the checking. The clamp was one constant, and it worked on the first try - but testing the truncation promise end-to-end, with answers big enough to hurt, is what surfaced a silent 4 KB limit that had been dropping legitimate large answers for years, in a codebase thousands of people run, where it looked like nothing at all: an occasional SERVFAIL on an unusual record, blamed on the upstream. The amplification work made the resolver honest. Verifying it is what made the resolver correct.

The ladder is closed. The remaining work on this resolver is no longer about amplification at all - but that is a different article.