The Handshake That Failed One Time in Twenty.

What the capture showed

We assumed it was our build. This nginx runs against BoringSSL rather than OpenSSL, with a patch of our own for ECH, so when something breaks in TLS the honest first guess is that we broke it. That guess was wrong and it cost most of a day before we stopped defending it.

The thing that moved it forward was giving up on logs and taking a packet capture. QUIC Initial packets are encrypted, but with keys derived from the connection ID, which travels in the header (RFC 9001, section 5.2). Unlike the rest of the connection they decrypt without a keylog file, and tshark will show the CRYPTO frames directly:

tshark -r fail.pcap -T fields -e ip.src -e quic.long.packet_type \
    -e quic.packet_number -e quic.crypto.offset -e quic.crypto.length

In a failing connection the server sends its ServerHello across two Initial packets - offset 0 length 1124, then offset 1124 length 54 - and one of the two never arrives. Which one does not matter: of the three captures we published, two lose the 54-byte tail and one loses the 1124-byte head, and all three deadlock identically. The client keeps sending Initial packets. The server answers with more Initial packets containing nothing but acknowledgements. Neither side ever sends the missing fragment again, and the connection sits there until it is closed.

Laid out packet by packet, from the debug log of one failing connection, it looks like this:

Two details in that ladder are worth pausing on. The client's acknowledgement is honest and useless at the same time: it reports exactly which packets arrived, and the gap it leaves is the one packet that can never be filled. And the probes the server does send go out at the Handshake level, while the data waiting to be acknowledged is still in the Initial packet number space - a probe in the wrong room.

Two things have to be true at once for this to happen, and both of them are recent:

1. The ServerHello has to be too big for one packet. A classical key share is about 120 bytes and fits comfortably. X25519MLKEM768, the post-quantum key exchange, carries 1088 bytes and pushes the ServerHello to roughly 1178, across the boundary.
2. The certificate flight has to span several Handshake packets. That keeps the probe timer anchored at the Handshake level while the oldest unacknowledged data is still in the Initial packet number space. With a single small self-signed certificate we could not reproduce it in 150 attempts; with a realistic three-certificate chain it happened reliably.

Both are ordinary for a public HTTP/3 site in 2026. That is why it was worth chasing: not because it was exotic, but because it was not.

Why the retransmission never left

nginx does detect the loss. The debug log says so plainly:

quic detect_lost pnum:2 thr:50 pthr:3 wait:-124 level:0
quic resend packet pnum:2

It calls ngx_quic_resend_frames() and puts the CRYPTO frame back on the queue. Then it never sends it. The reason is two lines apart, in different files.

When the loss is detected the congestion window collapses below the bytes already in flight:

quic congestion ack  idle t:... win:12000 if:6221   <- healthy
quic congestion lost      t:... win:3514  if:5021   <- window < in flight

and ngx_quic_output() passes that comparison down as a flag:

n = ngx_quic_output_packet(c, ctx, p, len, min,
                           cg->in_flight >= cg->window);

which becomes a filter that drops everything except acknowledgements:

if (ack_only && f->type != NGX_QUIC_FT_ACK) {
    break;
}

That is the "Initial packets carrying nothing but ACKs" from the capture, in source form.

The part that makes it a deadlock rather than a delay is that in_flight can never come down. The outstanding bytes are the Handshake packets carrying the certificate. The client cannot decrypt those, because deriving handshake keys requires the ServerHello, which is the frame being suppressed. So it cannot acknowledge them. So the window never reopens. So the frame is suppressed again, on every attempt, indefinitely.

Probe packets escape the filter, which is why the connection stays alive rather than dying immediately. ngx_event_quic_ack.c sets f->ignore_congestion = 1 on the PING frames it generates for a probe timeout. The CRYPTO frame requeued by ngx_quic_resend_frames() keeps the default of 0. That flag is set in three places in the tree and consulted in exactly one, ngx_quic_frame_sendto() - never in the main output loop.

Reporting it, and what came back

We wrote a reproduction before writing the report: two network namespaces, a veth pair at MTU 1500, netem adding 20 ms and 2% loss each way, a three-certificate RSA-4096 chain, and a loop firing HTTP/3 requests until one hung. About 5% failed. With ssl_ecdh_curve X25519 - classical, single-packet ServerHello - 100 out of 100 passed.

That bundle mattered more than the prose did. A maintainer who can run the thing in two minutes is in a completely different position from one who has to take your word for it.

nginx replied with a fix two days later, in ngx_quic_congestion_lost(). One line:

- cg->ssthresh = cg->in_flight * NGX_QUIC_CUBIC_BETA / 10;
+ cg->ssthresh = cg->w_prior * NGX_QUIC_CUBIC_BETA / 10;

Ours had been two hunks that let lost handshake CRYPTO bypass congestion control entirely. Theirs stops the window collapsing that far in the first place, deriving the new threshold from the window before the reduction rather than from whatever remained in flight after it, citing RFC 9438 section 4.6. It touches less, and it is a better change than the one we proposed. Maintainers generally prefer the change that touches less, and they were right to.

Rebuilding it against a different TLS library

The useful thing to do with someone else's fix is not to have an opinion about it. It is to build it and measure it, ideally somewhere the original verification did not run. Theirs used OpenSSL 3.5.7 on both ends. Ours is nginx 1.31.3 against BoringSSL, with a curl 8.21.0 and ngtcp2 1.23.0 client built against OpenSSL 4.0.1 - different implementations at each end of the connection.

On the reported case it cleared it completely:

Ten in 200 unpatched is the same rate the maintainer measured independently. The two remaining failures at 15% loss are ordinary timeouts rather than deadlocks: a build carrying both #1617 and our own change scores the same two. The bug and the fix both behave the same way across two different TLS stacks, which is the useful part of repeating a measurement someone else has already taken.

Two ideas we measured and threw away

Then we tried to break the fix, and got it wrong twice. Both are here because they were the two most interesting-looking results we got, and both were false.

The first wrong idea: a cascade. During the deadlock nothing can be acknowledged, so in_flight is frozen. We reasoned that repeated loss events would therefore shrink the window geometrically past it - 12000, 8400, 5880, 4116 - and the third reduction would re-open the deadlock even with the fix in place. We ran it at 8% loss expecting failures and got zero in 150. The reason is a guard we had not read carefully enough:

timer = f->send_time - cg->recovery_start;

if ((ngx_msec_int_t) timer <= 0) {
    goto done;
}

A whole burst of losses from one flight causes exactly one reduction, not a cascade. That is RFC 9002 behaving correctly. Our idea required three separate retransmissions each to be lost in turn, which essentially never happens.

The second wrong idea: a regression. At a larger certificate size we measured 29 failures against stock's 16 and briefly believed the fix made things worse. It does not. Repeat runs put the two level, and at 300 attempts the fix came out slightly ahead of stock. Sixteen against twenty-nine looks like a finding and is actually a sample size.

Finding both of those out privately cost about an hour. Posting either one upstream would have cost a maintainer five minutes to disprove, and cost us something harder to get back.

What the fix depends on

The third idea was right, and it came from asking why the fix works at all rather than whether it does.

nginx's initial congestion window is a fixed 12000 bytes, in ngx_event_quic.c. The reported case runs at win:12000 if:6221 - plenty of headroom. Taking 70% of 12000 still leaves 8400, comfortably above the 5021 bytes outstanding, so the CRYPTO frame leaves. That headroom is the entire mechanism.

So: what happens when the flight fills the window? We rebuilt the chain large enough that it did, changed nothing else on the stand, and ran it again.

The 29 in that third row is the second wrong idea, left in rather than tidied away: it is variance, not a regression, and the 300-attempt row above it is the one to trust.

The debug log shows why the fix stops helping, and the numbers land exactly where the arithmetic says they should:

The fix raises the window by about 800 bytes, exactly as designed. It needs about 2800 more. Below in_flight, the ACK-only filter closes again and the CRYPTO frame is dropped exactly as before - same signature, same silence, with in_flight climbing (12193, 12261, 12329) as probe PINGs are added on top:

quic congestion lost t:... win:9240 if:12057
quic frame tx hs:11 PING
quic frame tx init:5 ACK n:0 delay:0 5-4
quic frame tx init:6 ACK n:0 delay:0 6-4

A build carrying both #1617 and a check for ignore_congestion in the output loop reaches byte-for-byte the same congestion state and sends the retransmission:

quic congestion lost t:... win:9240 if:12057
quic frame tx init:6 CRYPTO len:54 off:1124

Same numbers, one boolean of difference. Across the debug runs, every connection that hit a collapse deadlocked under both stock and #1617 - six of six in each - and none did with the output-loop check, zero of seven.

The migration that caused this also removes the headroom

A 10 KB certificate flight is not normal today, and it is worth being precise about that rather than implying otherwise. Three RSA-4096 certificates come to about 3.9 KB. A hundred subject alternative names, the usual cap a CA will issue, adds roughly 2.5 KB more, so a large classical chain lands near 6.4 KB and stays under the window. The 300-SAN leaf we used to reach 10.5 KB is a size device, not a realistic certificate.

Post-quantum certificates are not a device. ML-DSA-44 has a 1312-byte public key and a 2420-byte signature (FIPS 204), so a three-certificate chain is roughly 12.4 KB. ML-DSA-65 is nearer 16 KB. On this stand in_flight lands at about the DER chain size plus 1.6 KB, which puts the threshold at roughly a 10.4 KB chain.

Which makes the choice of signature algorithm, rather than the decision to go post-quantum at all, the thing that decides which side of the line a server lands on:

Two caveats on that table. Falcon's signatures are compressed and therefore variable, around 666 bytes on average against a 752-byte ceiling, and FN-DSA is still a draft rather than a published standard - it is in the table because it is the one lattice signature small enough to keep a chain under the window, not because anyone is issuing certificates with it today. And every chain figure except RSA-4096 is arithmetic on published key and signature sizes plus a few hundred bytes of names and extensions per certificate, not something we measured.

ML-DSA is what will actually arrive first, because it is what has been standardised and what certificate authorities will issue. But it is not the end of the list. NIST is running a second process for additional signature schemes, and its Round 2 candidates are spread across a much wider size range than the lattice signatures already standardised:

None of these is standardised, all of them are Round 2 candidates rather than finalists, parameter sets are still moving, and the figures are arithmetic on submission documents rather than anything we measured. Treat the table as a range, not a forecast.

The two MAYO rows are the same algorithm at two parameter sets, and they land on opposite sides of the line. That is worth more than any single number here. A scheme with a tiny signature and a large public key looks efficient right up until you remember where public keys live: every certificate on the wire carries its own. Only the root's key is absent, because it is already a local trust anchor. So the small-signature trade buys nothing on a chain and can cost a great deal, which is how MAYO_2 ends up further into the deadlock zone than ML-DSA-65.

Size is also not the only axis. SQIsign's chain is under 2 KB and would never come close to this failure, but its verification is orders of magnitude slower than a lattice signature, which is its own problem for a resolver completing many handshakes a second. A scheme that is kind to your congestion window can be unkind to your CPU. The point is not that one of these is right, only that the window question has a different answer for every one of them, and it is worth asking before the certificate arrives rather than after.

This generalises past nginx. Any QUIC implementation with a fixed initial congestion window and a rule about what may be sent once that window is exhausted has the same arithmetic to check, and the check is small: take your initial window, multiply by your reduction factor, and compare the result against the bytes a full server flight puts in flight. If the second number is larger, a single lost Initial packet is enough.

Reproduce it yourself

None of this has to be taken on trust. The bundle is public and it is the same one the maintainers were given: github.com/Ozy-666/nginx-quic-initial-repro. It needs root for namespaces and netem, an nginx built --with-http_v3_module against a TLS library offering X25519MLKEM768, and an HTTP/3-capable curl.

git clone https://github.com/Ozy-666/nginx-quic-initial-repro
cd nginx-quic-initial-repro

./make-certs.sh          # 3-cert RSA-4096 chain, ~3.9 KB DER
./setup-netns.sh         # srv+cli namespaces, MTU 1500, 20 ms + 2% loss
mkdir -p html logs tmp && echo ok > html/index.html
ip netns exec srv nginx -c "$PWD/nginx.conf.example" -p "$PWD"
./run.sh 150

Expect roughly 4 to 5% of attempts to hang until the client's handshake timeout. To confirm the diagnosis rather than just the symptom, set ssl_ecdh_curve X25519; and run it again: the ServerHello fits in one packet and 100 out of 100 succeed.

For the window-filling case, build the larger chain and point ssl_certificate at it:

./make-bigchain.sh pki-big 300     # ~10.5 KB DER

What we would tell ourselves at the start

Stop defending your own build. We lost hours to "it is probably our patch" because that was the humble assumption and humility felt like rigour. It is not. It is just another untested hypothesis, and the fact that it is unflattering does not make it more likely to be true.

Take the capture earlier. The logs said nothing for a day. The capture said almost everything in ten minutes, because QUIC Initial packets decrypt without a keylog and nobody remembers that until they need it.

A reproduction is worth more than an analysis. The analysis in our report may well have been ignored. What produced a fix in two days was a script that made the failure happen on someone else's machine.

Test the clever thought before you publish it. Two of ours were wrong, and they were the two that felt most like findings.

The general form of the bug is worth carrying away even if you never touch QUIC. Two subsystems were each behaving correctly on their own terms: one decided a frame must be retransmitted, the other decided nothing may be sent. Neither is wrong, neither logs an error, and the failure lives entirely in the assumption they do not share - that something in flight will eventually be acknowledged. Wherever a rate limit governs the only message that could lift the rate limit, the same deadlock is available.