Post-mortem: the QUIC Handshake That Failed One Time in Twenty

DoH3 on this resolver failed about one connection in twenty and logged nothing. The cause is in nginx: when a post-quantum ServerHello spans two QUIC Initial packets and one is lost, the loss is detected and the CRYPTO frame is requeued, but the congestion window has collapsed below the bytes in flight and the output loop sends acknowledgements only. The client cannot derive handshake keys without that frame, so it cannot acknowledge anything, so the window never reopens. Reported as nginx/nginx#1616 with a scripted reproduction; nginx proposed a one-line fix, #1617, which clears the reported case on a different TLS library than the one used to verify it. Extending the test found the limit of that fix: a certificate flight large enough to fill the 12000-byte initial window deadlocks again, and post-quantum certificates are that large.

A new post-mortem documents an nginx bug found on this resolver's own DoH3 endpoint, and what happened after reporting it.

The symptom was silence. Roughly one HTTP/3 connection in twenty hung until the client's timeout, with nothing logged at warn or error, and a retry usually succeeded. A packet capture ended the guessing: QUIC Initial packets are encrypted with keys derived from the connection ID, so they decrypt without a keylog file. In a failing connection the ServerHello arrives in two Initial packets, offset 0 length 1124 and offset 1124 length 54, and the second one never comes back.

nginx detects the loss and requeues the frame. Its debug log says so. What stops the retransmission is two lines apart, in different files: on loss the congestion window collapses below the bytes already in flight, and the output loop is then told to send acknowledgements only, skipping every other frame. The outstanding bytes are the Handshake packets carrying the certificate, which the client cannot decrypt or acknowledge until it has the ServerHello that is being suppressed. The window can never reopen. Congestion control ends up protecting the network from the one frame that would end the congestion.

Two conditions have to hold together, and both are recent. A post-quantum key share pushes the ServerHello past a single packet, and a realistic certificate chain keeps the probe timer anchored at the Handshake level. Both are ordinary for a public HTTP/3 site.

We reported it as nginx/nginx#1616 with a scripted reproduction in two network namespaces. nginx replied two days later with a one-line change, #1617, deriving the reduced threshold from the window before the reduction rather than from what remained in flight after it. It is a smaller change than the one we proposed and a better one. Rebuilt here against BoringSSL, with an OpenSSL-based client, it cleared the reported case: 10 failures in 200 unpatched, 0 in 200 with it.

Extending the test found where it stops working. The fix depends on headroom: nginx's initial congestion window is a fixed 12000 bytes and the reported case only puts 6221 in flight, so a reduction still leaves room. With a certificate flight large enough to fill that window the reduction lands below the bytes outstanding, the acknowledgement-only filter closes again, and the handshake deadlocks exactly as before. A classical chain does not reach that size. Three ML-DSA-44 certificates are about 12.4 KB and three ML-DSA-65 about 16 KB, which do. The migration that caused this bug and the migration that defeats its fix are the same migration.

The write-up includes two hypotheses we measured and discarded, because both looked like findings and both were wrong. Both issues are open, not merged, at the time of publication.

Highlights