Running, and Answering Nothing: a UDP Listener That Died Without a Log

What it looked like

Twelve identical queries to the resolver's own loopback address returned four answers and eight timeouts. Repeating the test gave a different four. The pattern was stable in proportion and random in which queries survived, which is the signature of a load-balancing decision rather than a failing dependency - a broken upstream fails every query, not three out of four.

Everything else was fine. The encrypted transports were unaffected throughout, the web interface was responsive, the upstream was reachable, and systemctl reported the unit active (running) with no restarts. A monitoring system that checks "is the process up" and "does the API respond" would have reported a perfectly healthy resolver for as long as the condition lasted, which in our case was roughly seventeen hours.

The code path

The read loop is eight lines, and the whole failure is in one of them. This is proxy/serverudp.go in dnsproxy, the DNS engine inside AdGuard Home and a number of other tools:

for p.isStarted() {
    n, localIP, remoteAddr, err := proxynetutil.UDPRead(conn, b, p.udpOOBSize)
    if n > 0 {
        // ... handle the query ...
    }

    if err != nil {
        logUDPConnError(err, conn, p.logger)

        break        // loop ends here, forever; conn is never closed
    }
}

And the logging it does on the way out:

func logUDPConnError(err error, conn *net.UDPConn, l *slog.Logger) {
    if errors.Is(err, net.ErrClosed) {
        l.Debug("udp connection closed", "addr", conn.LocalAddr())  // off by default
    } else {
        l.Error("reading from udp", slogutil.KeyError, err)
    }
}

Three properties compound here, and none of them is dangerous alone.

There is no recovery. One goroutine is started per listener and nothing supervises it. A loop that returns is not replaced for the lifetime of the process.

The socket is not closed. This is the part that turns a dead goroutine into dropped traffic, and we come back to it below.

It can be completely silent. One of the two error classes logs at Debug, and AdGuard Home ships with verbose: false. Nothing in the file increments a counter or exports a metric, so no health check, dashboard or log line can tell "listener dead" apart from "quiet period".

Why not closing the socket is the expensive part

If the loop had closed its socket on the way out, the outcome would have been ugly but honest. On a single-socket setup the port would have gone away and clients would have received ICMP port-unreachable - an immediate, visible failure that any operator can diagnose in a minute. On a sharded setup the socket would have left the SO_REUSEPORT group and the kernel would have redistributed its share across the surviving readers, with no loss at all.

Leaving it bound produces the worst of both. The socket stays in the group, the kernel goes on hashing its share of datagrams into a receive buffer that nothing drains, and once that buffer is full every subsequent datagram for that socket is discarded in the kernel. The client sees no answer and no rejection, so it retries until it times out. From the outside this is indistinguishable from packet loss on the network.

That is the general shape worth taking away from this article, and it is not specific to Go, to DNS, or to this project: a bound socket with no reader is a black hole, and a closed socket is an error message. When a serving path gives up, closing its socket is not cleanup, it is the notification.

Where the kernel tells you, if you ask

The application said nothing. The kernel had been counting the whole time. Two commands show it, and both work against any UDP service, not just a DNS server.

The first is ss with -m, which prints socket memory. Run it against the port you care about:

ss -lunpm 'sport = :53'

Here is the healthy output from this resolver, taken while writing this article. Four sockets share the port, all four are being drained:

State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port  Process
UNCONN  0       0                   *:53                *:*     users:(("AdGuardHome",pid=1867048,fd=11))
   skmem:(r0,rb8388608,t0,tb1048576,f4096,w0,o0,bl0,d0)
UNCONN  0       0                   *:53                *:*     users:(("AdGuardHome",pid=1867048,fd=16))
   skmem:(r0,rb8388608,t0,tb1048576,f4096,w0,o0,bl0,d0)
UNCONN  0       0                   *:53                *:*     users:(("AdGuardHome",pid=1867048,fd=17))
   skmem:(r0,rb8388608,t0,tb1048576,f4096,w0,o0,bl0,d0)
UNCONN  0       0                   *:53                *:*     users:(("AdGuardHome",pid=1867048,fd=18))
   skmem:(r0,rb8388608,t0,tb1048576,f4096,w0,o0,bl0,d0)

Three fields carry the whole diagnosis:

During the incident, three of the four sockets sat with Recv-Q exactly at rb and their d counters rising, while the fourth sat at zero. That single screen is the entire fault, visible in one command, at any point during the seventeen hours nobody knew to run it.

The second place to look is the kernel's own UDP counters:

grep -A1 '^Udp:' /proc/net/snmp

Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors ...
Udp: 22515772    270929  2021423  22623187     2021423      0

RcvbufErrors is the count of datagrams the kernel threw away because a socket's receive buffer was full. There is one trap in reading it, and it is worth stating plainly because we nearly fell into it ourselves.

So sample it instead of reading it. Over a 60-second window on the repaired resolver, InDatagrams rose by 7,943 and RcvbufErrors did not move at all - it stayed at 2,021,423 across all seven samples. That is what a healthy socket looks like on a counter that still carries a bad day:

# print RcvbufErrors every 10 seconds; watch for movement, not the total
for i in $(seq 1 7); do
  printf "%s %s\n" "$(date +%T)" \
    "$(awk '/^Udp:/{if(h)print $6; else h=1}' /proc/net/snmp)"
  sleep 10
done

Reproduce the consequence in 60 seconds

We never reproduced the error that triggered the exit - more on that below. The consequence, though, needs no bug at all, and it is the half that makes the failure severe. Bind a UDP socket, never read from it, and send it traffic. Nothing here touches a running service; it uses a high port on loopback:

# a socket that binds and never reads
python3 - <<'EOF' &
import socket, time
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 4096)
s.bind(("127.0.0.1", 15399))
time.sleep(20)
EOF

# 50,000 datagrams at it
python3 -c "
import socket, time
time.sleep(1)
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
for _ in range(50000): s.sendto(b'x'*512, ('127.0.0.1', 15399))
print('sent 50000 datagrams')"

ss -lunpm 'sport = :15399'

The real output from that run:

sent 50000 datagrams
State   Recv-Q  Send-Q  Local Address:Port    Peer Address:Port  Process
UNCONN  8960    0           127.0.0.1:15399        0.0.0.0:*     users:(("python3",pid=3413433,fd=3))
   skmem:(r8960,rb8192,t0,tb1048576,f3328,w0,o0,bl0,d49993)

49,993 of 50,000 discarded, and the process that owns the socket is running normally the entire time. It has no idea any of this happened - there is no callback, no signal and no error for "the kernel dropped a datagram addressed to you". The only record is the d counter you just read.

One incidental detail, since it will show up in your own output: we asked for a 4,096-byte buffer and rb reports 8,192. The kernel doubles the requested size to cover its own bookkeeping overhead. Recv-Q reading 8,960 rather than exactly 8,192 is the same accounting, not an overflow.

What we could not determine

We cannot say which read error triggered the exit, because nothing logged it. That sounds like a gap in the investigation, and it is - but it is also the finding itself, so it is worth being precise about what was ruled out rather than glossing it.

Not semaphore exhaustion. The request semaphore is shared across all the loops, and the surviving loop kept acquiring and releasing slots throughout, so slots were available. The other break in that function, the one after the semaphore acquire, logs at Error. No such line exists.

Not shutdown. p.isStarted() is the same flag for every loop, and the surviving loop kept running on it.

Not a lost log line. The journal showed no rate-limiting or suppression for the unit in that period.

What remains is the read-error break, most plausibly through the net.ErrClosed branch that logs at debug level. "Most plausibly" is as far as the evidence reaches, and we said exactly that upstream rather than naming a cause we could not show. A failure mode that destroys its own evidence cannot be diagnosed by the person it happens to, which is the argument for fixing the silence whether or not anyone ever identifies the trigger.

The fix, and what it is careful about

The read loop needs to distinguish three cases instead of treating every error the same way. We implemented this as a pure predicate with a table test, so the policy is readable in one place and testable without a socket:

The reopen is the obvious half. The close is the half that matters, and it is worth separating the two: even if no replacement can be opened, closing removes the dead socket from the SO_REUSEPORT group so the kernel stops delivering to a reader that will never come. On a single-socket deployment, closing converts a silent black hole into a connection refusal. Neither of those is a good day, but both are diagnosable in the first minute rather than the seventeenth hour.

Independently of any recovery policy, the exit is now logged at Error unconditionally and counted. A code path that stops serving queries should never be quiet, regardless of which error class got it there. That was the part we cared most about: the recovery is a convenience, the log line is the thing that would have saved seventeen hours.

This is running here. The resolver was rebuilt on 14 September and the fix has been in the serving path since 08:09 on 15 September, which is where the healthy ss output above comes from.

Reported upstream, and open

The report is AdguardTeam/dnsproxy issue 525, filed on 13 September 2026 with the code path, the measurements, the reproduction above and a link to our implementation as a starting point if it is useful in that shape. As of the day this article was published it is open, labelled P4 and has no comments. We will note the outcome here when there is one, whatever it is - including if the maintainers decide the current behaviour is intentional and we have misread it.

Being specific about the scope: this affects the UDP listener only. DoH, DoT and DoQ each run their own accept paths and were healthy throughout, which is exactly why the failure was so quiet - on most installations plain UDP on port 53 is what every device on the network uses, and it is the one transport with no connection state for anything to notice going wrong. The multiplexed transports fail differently rather than better: tearing down one of those connections tears down every query riding on it, which is the shape behind two of the nine use-after-frees fixed in Unbound 1.26.1, one for DoH and one for DoQ.

Check your own resolver

None of this is specific to one project. Any UDP service can be left in this state by any read loop that gives up without closing. Three checks, in the order we would run them:

# 1. any socket on :53 with a non-zero drop counter?
ss -lunpm 'sport = :53' | grep -E 'Recv-Q|skmem'

# 2. is RcvbufErrors moving? (total alone means nothing)
awk '/^Udp:/{if(h)print $6; else h=1}' /proc/net/snmp

# 3. does it actually answer? twelve times, not once
for i in $(seq 1 12); do
  dig +short +time=2 +tries=1 @127.0.0.1 example.com A >/dev/null \
    && echo ok || echo TIMEOUT
done | sort | uniq -c

The third one is the check most monitoring misses, and the reason to run it twelve times rather than once is the whole point of this article. A single query has a three-in-four chance of landing on a dead socket in our case and a one-in-four chance of looking fine. Any probe that sends one packet and calls the service healthy will report whichever answer it happened to get. If you take one operational habit from this: a health check that sends a single query cannot detect a partial outage, and partial is how this kind of failure arrives.

If you want the same confidence about the encrypted transports rather than plain UDP, the per-protocol checks are in how to test whether your DNS is really encrypted, and the difference between "my resolver answered" and "my resolver is the one I configured" is covered in what a DNS resolver actually is.

The general lesson

Every serving path has an exit, and the exits are where the observability debt collects. The happy path in this loop is instrumented, metered and logged. The line that ends the loop forever had one debug statement and no counter, because at the time it was written it was understood as a shutdown path, and shutdown paths do not need to be loud.

Two things follow. The first is a code review question worth asking routinely: what happens to the socket when this loop returns? If the answer is "it stays bound", the loop cannot simply return. The second is a monitoring question: a process being alive, a port being bound and an API responding are three facts that say nothing about whether queries are being answered. Only a query says that, and only several of them say it reliably.

The fault here lasted seventeen hours and was visible the entire time in a single line of ss output. Nobody was looking, because nothing suggested there was anything to look for. That is the cost of a quiet exit, and it is paid by whoever is running the software, not by whoever wrote it.