The Handshake That Failed One Time in Twenty.

What the capture showed

We assumed it was our build. This nginx runs against BoringSSL rather than OpenSSL, with a patch of our own for ECH, so when something breaks in TLS the honest first guess is that we broke it. That guess was wrong and it cost most of a day before we stopped defending it.

The thing that moved it forward was giving up on logs and taking a packet capture. QUIC Initial packets are encrypted, but with keys derived from the connection ID, which travels in the header (RFC 9001, section 5.2). Unlike the rest of the connection they decrypt without a keylog file, and tshark will show the CRYPTO frames directly:

tshark -r fail.pcap -T fields -e ip.src -e quic.long.packet_type \
    -e quic.packet_number -e quic.crypto.offset -e quic.crypto.length

It is worth seeing what that returns, because the whole investigation turns on it. First a handshake that works, captured on the same rig with the loss switched off:

tshark -r captures/ok.pcap \
  -Y 'ip.src==10.66.0.1 && quic.long.packet_type==0 && quic.crypto.length' \
  -T fields -e quic.packet_number -e quic.crypto.offset -e quic.crypto.length

1    0      1124
2    1124   54

Two frames, and the offsets meet exactly: 0 plus 1124 is 1124, so the second packet carries the remaining 54 bytes of a 1178-byte ServerHello. After those two, the server sends no further Initial packets at all. The handshake moves on.

The identical command on a failing capture:

# ... same filter, r17.pcap

1    0      1124
# and nothing else, for the life of the connection

One frame where there should be two. The 54-byte tail is absent - not late, not reordered, absent. Nine more Initial packets follow it over the next eight seconds, at exponentially widening intervals, and not one of them carries a CRYPTO frame.

And on a third capture, the opposite half is missing:

2    1124   54

One detail in that table corrects something we wrote in the original report. The Initial packets following the ServerHello are not purely acknowledgements: in r17 and r42 two of them are PING probes issued at the Initial level, which is the packet number space holding the unacknowledged data. The connection we captured with --with-debug probed only at the Handshake level, sixteen times. So probe placement varies between connections, and the tidy claim that nginx always probes the wrong space is not one the evidence supports. It does not matter to the outcome: a PING carries no CRYPTO frame, so probing the right space does not deliver the missing 54 bytes either.

In a failing connection the server sends its ServerHello across two Initial packets - offset 0 length 1124, then offset 1124 length 54 - and one of the two never arrives. Which one does not matter: of the three captures we published, two lose the 54-byte tail and one loses the 1124-byte head, and all three deadlock identically. The client keeps sending Initial packets. The server answers with more Initial packets, carrying acknowledgements and probes, and never another CRYPTO frame. Neither side ever sends the missing fragment again, and the connection sits there until it is closed.

Laid out packet by packet, from the debug log of one failing connection, it looks like this. The first three rows come from the captures rather than the log: quic_retry is on, so the connection nginx logs is the one after address validation, and it opens with a ClientHello that already carries a token.

Two details in that ladder are worth pausing on. The client's acknowledgement is honest and useless at the same time: it reports exactly which packets arrived, and the gap it leaves is the one packet that can never be filled. And every probe in this connection went out at the Handshake level, while the data waiting to be acknowledged was still in the Initial packet number space. That is not universal - two of the captures show Initial-level probes - but it makes no difference either way, because a probe carries a PING, never the CRYPTO frame that is missing.

Two things have to be true at once for this to happen, and both of them are recent:

1. The ServerHello has to be too big for one packet. A classical key share is about 120 bytes and fits comfortably. X25519MLKEM768, the post-quantum key exchange, carries 1088 bytes and pushes the ServerHello to roughly 1178, across the boundary.
2. The certificate flight has to span several Handshake packets. It gives the server something to keep in flight that the client cannot acknowledge, which is what holds the congestion window shut. With a single small self-signed certificate we could not reproduce it in 150 attempts; with a realistic three-certificate chain it happened reliably.

Both are ordinary for a public HTTP/3 site in 2026. That is why it was worth chasing: not because it was exotic, but because it was not.

The same failure from the client's side

Everything above is the server's account of it, taken from a debug build nobody else is running. An operator meeting this in the wild has none of that. They have this:

$ curl -sS --http3-only https://example.com:8443/
curl: (55) Failed to connect to example.com:8443 after 10001 ms:
Failed sending data to the peer

The message names neither QUIC, nor the handshake, nor a cause. The connection is not refused and DNS is fine; TLS never reports an error because TLS never gets far enough to have an opinion. Ten seconds pass and the request dies. In a browser it is a spinner and then a generic failure page, with a working site one reload later.

We had never looked at our own bug from that end. On 30 August 2026 we re-ran the public bundle with client-side qlog capture switched on - ngtcp2 writes one file per connection when QLOGDIR is set - and took 80 attempts, of which six failed. That is 7.5%, in the same range as the 4 to 6% measured in the runs above. They do not all look the same, and the way they differ turned out to matter more than we realised at the time. Here is one of them, folded to the packets that carry the story:

    ms  dir  packet     contents
     0  TX   initial    ClientHello, 1521 B
    41  RX   retry
    41  TX   initial    ClientHello again, post-Retry
    86  RX   initial    CRYPTO off=1124 len=54   <- the tail, and no ACK with it
    86  TX   initial    ACK
  1041  TX   initial    CRYPTO off=0..1521       <- PTO probe carrying the ClientHello again
  1082  RX   initial    ACK
  1183  TX   initial    PING       -> RX ACK
  1396  TX   initial    PING       -> RX ACK
  1735  TX   initial    PING       -> RX ACK
  2304  TX   initial    PING       -> RX ACK
  3300  TX   initial    PING       -> RX ACK
  5095  TX   initial    PING       -> RX ACK
  8412  TX   initial    PING       -> RX ACK
 10001       (curl gives up at its handshake deadline)

In that connection the client receives exactly one CRYPTO frame and it starts at offset 1124: the 54-byte tail of the ServerHello, with the 1,124 bytes that should precede it missing. From there it probes on the widening PTO schedule visible in the timestamps, and every probe comes back as a bare ACK. The connection stays alive and useless for ten seconds, both ends behaving correctly by their own lights, until the client's handshake deadline ends it.

The six failures split two ways, and the split is not cosmetic. Four of them receive the 1,124-byte head, in a packet that also carries an acknowledgement of their ClientHello. Two receive only the 54-byte tail, in a packet carrying no acknowledgement at all. That difference decides what the client does next, and it follows from RFC 9002 section 6.2.4 rather than from anything either implementation chose:

A client with nothing unacknowledged has nothing to put in a probe, so it sends a PING. A client still holding unacknowledged data puts that in the probe instead, and the acknowledgement ranges show each one re-sending exactly what was outstanding and nothing more. Every client here splits its 1,521-byte ClientHello across two Initial packets:

5abd166b - head arrives
   87ms RX pn=1  ACK ranges=[[2, 3]] + CRYPTO off=0 len=1124
  194ms TX pn=5  PING                        <- both Initials acked, nothing to carry

fe074270 - tail arrives
   82ms RX pn=0  ACK ranges=[[2]]            <- covers pn=2 only, not pn=3
   87ms RX pn=2  CRYPTO off=1124 len=54
  163ms TX pn=5  CRYPTO 998..1521 + PING     <- exactly the 523 B still unacked

753d74b6 - tail arrives
   86ms RX pn=2  CRYPTO off=1124 len=54      <- no ACK received at all
 1041ms TX pn=5  CRYPTO 0..1521 + PING       <- the whole ClientHello again

So the mechanism is the acknowledgement, not the fragment as such: the packet carrying the 1,124-byte head happened also to carry the ACK that covered both of the client's Initials, and the packets carrying the 54-byte tail carried no ACK with them. fe074270 had received a separate ACK five milliseconds earlier, but it covered only one of its two Initial packets, and the 523 bytes it re-sent at 163 ms are precisely the range that one left out. Once the server acknowledges the re-send, both fall back to bare PINGs, which is why the tails of these connections look like the other four.

It is not a decryption failure, and that matters

The natural way to describe a handshake that dies like this is that the client could not decrypt it, and that is the phrase people reach for when they go looking for an answer. The qlog says otherwise, twice over: across all six failures not one Handshake-level packet is ever received, and ngtcp2 logs no dropped, discarded or undecryptable packet at any point in any of them.

There is nothing to decrypt. Initial packets are protected with keys derived from the connection ID, which is why the capture in the previous section is readable at all; Handshake keys come from the TLS material carried inside the CRYPTO frames, and the fragment carrying it is the one that went missing. The client never reaches the point of holding a key that could fail. Saying it "failed to decrypt" would put the fault at the last step of a sequence that stopped at the first.

The distinction is practical, not pedantic. A genuine decryption failure means packets arrived and could not be read - a key mismatch, a broken middlebox, a version problem - and the fix is somewhere in the crypto. This is the opposite shape: nothing arrived, everything that did arrive was read correctly, and the fix is in retransmission logic.

Three fingerprints, none of which need server access

• It dies at the deadline, not on an error. Ten seconds for curl, and the number is the same on every failure to the millisecond. A handshake that fails for a real reason fails fast.
• The CRYPTO stream has a hole in it. One frame where there should be two, either the head with no tail or the tail with nothing before it. In a healthy handshake on the same rig the offsets meet exactly: 0 then 1124.
• Probe and ACK to the end. Both ends keep talking. Nothing times out at the network layer, nothing resets, and the traffic is real - which is why this is invisible to any monitoring that watches for errors rather than for completions.

The qlog capture costs one environment variable, and it is the fastest way we know to tell this apart from the many other ways a QUIC handshake can end badly:

$ QLOGDIR=./qlog curl -sS --http3-only https://your-host/
# one .sqlog file per connection; the failing one is the long one

Why the retransmission never left

nginx does detect the loss. The debug log says so plainly:

quic detect_lost pnum:2 thr:50 pthr:3 wait:-124 level:0
quic resend packet pnum:2

It calls ngx_quic_resend_frames() and puts the CRYPTO frame back on the queue. Then it never sends it. The reason is two lines apart, in different files.

When the loss is detected the congestion window collapses below the bytes already in flight:

quic congestion ack  idle t:... win:12000 if:6221   <- healthy
quic congestion lost      t:... win:3514  if:5021   <- window < in flight

and ngx_quic_output() passes that comparison down as a flag:

n = ngx_quic_output_packet(c, ctx, p, len, min,
                           cg->in_flight >= cg->window);

which becomes a filter that drops everything except acknowledgements:

if (ack_only && f->type != NGX_QUIC_FT_ACK) {
    break;
}

That is the run of Initial packets with no CRYPTO frame in them, from the captures above, in source form.

The part that makes it a deadlock rather than a delay is that in_flight can never come down. The outstanding bytes are the Handshake packets carrying the certificate. The client cannot decrypt those, because deriving handshake keys requires the ServerHello, which is the frame being suppressed. So it cannot acknowledge them. So the window never reopens. So the frame is suppressed again, on every attempt, indefinitely.

Probe packets escape the filter, which is why the connection stays alive rather than dying immediately. ngx_event_quic_ack.c sets f->ignore_congestion = 1 on the PING frames it generates for a probe timeout. The CRYPTO frame requeued by ngx_quic_resend_frames() keeps the default of 0. That flag is set in three places in the tree and consulted in exactly one, ngx_quic_frame_sendto() - never in the main output loop.

Reporting it, and what came back

We wrote a reproduction before writing the report: two network namespaces, a veth pair at MTU 1500, netem adding 20 ms and 2% loss each way, a three-certificate RSA-4096 chain, and a loop firing HTTP/3 requests until one hung. About 5% failed. With ssl_ecdh_curve X25519 - classical, single-packet ServerHello - 100 out of 100 passed.

That bundle mattered more than the prose did. A maintainer who can run the thing in two minutes is in a completely different position from one who has to take your word for it.

A candidate fix appeared two days later as nginx/nginx#1617, opened by a contributor with no prior association with the repository rather than by an nginx maintainer. It was one line in ngx_quic_congestion_lost():

- cg->ssthresh = cg->in_flight * NGX_QUIC_CUBIC_BETA / 10;
+ cg->ssthresh = cg->w_prior * NGX_QUIC_CUBIC_BETA / 10;

Ours had been two hunks that let lost handshake CRYPTO bypass congestion control entirely. That one stopped the window collapsing so far in the first place, deriving the new threshold from the window before the reduction rather than from whatever remained in flight after it, citing RFC 9438 section 4.6. It touched less than ours did.

On 4 September 2026 that branch was force-pushed and the CUBIC line is gone. The pull request now carries a different change under the same number, described below. If you read this article before that date and went to look at the code, what you found there was not what you found here, and this section is why.

Rebuilding it against a different TLS library

The useful thing to do with someone else's fix is not to have an opinion about it. It is to build it and measure it, ideally somewhere the original verification did not run. Their own verification ran OpenSSL at both ends. Ours is nginx 1.31.3 against BoringSSL, with a curl 8.21.0 and ngtcp2 1.23.0 client built against OpenSSL 4.0.1 - different implementations at each end of the connection.

On the reported case it cleared it completely:

Ten in 200 unpatched is 5%, matching our own earlier runs on three other setups. A maintainer reproduced the bug independently on 4 August but published no failure rate, so there is no second number to compare against, only a second confirmation that it happens. The two remaining failures at 15% loss are ordinary timeouts rather than deadlocks: a build carrying both the CUBIC change and our own scores the same two. The bug and the fix both behave the same way across two different TLS stacks, which is the useful part of repeating a measurement someone else has already taken.

Two ideas we measured and threw away

Then we tried to break the fix, and got it wrong twice. Both are here because they were the two most interesting-looking results we got, and both were false.

The first wrong idea: a cascade. During the deadlock nothing can be acknowledged, so in_flight is frozen. We reasoned that repeated loss events would therefore shrink the window geometrically past it - 12000, 8400, 5880, 4116 - and the third reduction would re-open the deadlock even with the fix in place. We ran it at 8% loss expecting failures and got zero in 150. The reason is a guard we had not read carefully enough:

timer = f->send_time - cg->recovery_start;

if ((ngx_msec_int_t) timer <= 0) {
    goto done;
}

A whole burst of losses from one flight causes exactly one reduction, not a cascade. That is RFC 9002 behaving correctly. Our idea required three separate retransmissions each to be lost in turn, which essentially never happens.

The second wrong idea: a regression. At a larger certificate size we measured 29 failures against stock's 16 and briefly believed the fix made things worse. It does not. Repeat runs put the two level, and at 300 attempts the fix came out slightly ahead of stock. Sixteen against twenty-nine looks like a finding and is actually a sample size.

Finding both of those out privately cost about an hour. Posting either one upstream would have cost a maintainer five minutes to disprove, and cost us something harder to get back.

What the fix depends on

The third idea was right, and it came from asking why the fix works at all rather than whether it does.

nginx's initial congestion window is a fixed 12000 bytes, in ngx_event_quic.c. The reported case runs at win:12000 if:6221 - plenty of headroom. Taking 70% of 12000 still leaves 8400, comfortably above the 5021 bytes outstanding, so the CRYPTO frame leaves. That headroom is the entire mechanism.

So: what happens when the flight fills the window? We rebuilt the chain large enough that it did, changed nothing else on the stand, and ran it again.

The 29 in that third row is the second wrong idea, left in rather than tidied away: it is variance, not a regression, and the 300-attempt row above it is the one to trust.

The debug log shows why the fix stops helping, and the numbers land exactly where the arithmetic says they should:

The fix raises the window by about 800 bytes, exactly as designed. It needs about 2800 more. Below in_flight, the ACK-only filter closes again and the CRYPTO frame is dropped exactly as before - same signature, same silence, with in_flight climbing (12193, 12261, 12329) as probe PINGs are added on top:

quic congestion lost t:... win:9240 if:12057
quic frame tx hs:11 PING
quic frame tx init:5 ACK n:0 delay:0 5-4
quic frame tx init:6 ACK n:0 delay:0 6-4

A build carrying both the CUBIC change and a check for ignore_congestion in the output loop reaches byte-for-byte the same congestion state and sends the retransmission:

quic congestion lost t:... win:9240 if:12057
quic frame tx init:6 CRYPTO len:54 off:1124

Same numbers, one boolean of difference. Across the debug runs, every connection that hit a collapse deadlocked under both stock and the CUBIC change - six of six in each - and none did with the output-loop check, zero of seven.

The migration that caused this also removes the headroom

A 10 KB certificate flight is not normal today, and it is worth being precise about that rather than implying otherwise. Three RSA-4096 certificates come to about 3.9 KB. A hundred subject alternative names, the usual cap a CA will issue, adds roughly 2.5 KB more, so a large classical chain lands near 6.4 KB and stays under the window. The 300-SAN leaf we used to reach 10.5 KB is a size device, not a realistic certificate.

Post-quantum certificates are not a device. ML-DSA-44 has a 1312-byte public key and a 2420-byte signature (FIPS 204), so a three-certificate chain is roughly 12.4 KB. ML-DSA-65 is nearer 16 KB. On this stand in_flight lands at about the DER chain size plus 1.6 KB, which puts the threshold at roughly a 10.4 KB chain.

Which makes the choice of signature algorithm, rather than the decision to go post-quantum at all, the thing that decides which side of the line a server lands on:

Three notes on that table. Falcon's signatures are compressed and therefore variable, around 666 bytes on average against a 752-byte ceiling, and FN-DSA has no published standard yet - FIPS 203, 204 and 205 were finalised in August 2024, FIPS 206 is still in preparation. It is in the table because it is the one lattice signature small enough to keep a chain under the window, not because anyone is issuing certificates with it. Every chain figure except RSA-4096 is arithmetic on published key and signature sizes plus a few hundred bytes of names and extensions per certificate, not something we measured.

And SLH-DSA is the row worth staring at. Its 32-byte public key is the smallest in the table and its signature is 17 KB, so a three-certificate chain is around 52 KB - more than four times the initial congestion window, and off the scale of the chart above entirely. The slower 128s variant halves the signature to 7856 bytes and still lands near 24 KB. Nothing about that is a defect: SLH-DSA's security rests on hash functions alone, which is exactly why it exists as the conservative fallback. It is simply not a TLS certificate algorithm, and this is one of the reasons why.

ML-DSA is what will actually arrive first, because it is what has been standardised and what certificate authorities will issue. But it is not the end of the list. NIST runs a separate process for additional signature schemes, explicitly looking for designs that avoid structured lattices and offer short signatures and fast verification. It reached Round 3 in May 2026, with nine candidates left, and they are spread across a far wider size range than anything already standardised:

None of these is standardised, none is a finalist - Round 3 is an evaluation round, not a shortlist for publication - and HAWK is in the table only for its sizes, because its submission team withdrew it from the process. Parameter sets are still moving and every figure is arithmetic on submission documents rather than anything we measured. Treat the table as a range, not a forecast.

The two MAYO rows are the same algorithm at two parameter sets, and they land on opposite sides of the line. That is worth more than any single number here. A scheme with a tiny signature and a large public key looks efficient right up until you remember where public keys live: every certificate on the wire carries its own. Only the root's key is absent, because it is already a local trust anchor. So the small-signature trade buys nothing on a chain and can cost a great deal, which is how MAYO_2 ends up further into the deadlock zone than ML-DSA-65.

Size is also not the only axis. SQIsign's chain is under 2 KB and would never come close to this failure, but its verification is orders of magnitude slower than a lattice signature, which is its own problem for a resolver completing many handshakes a second. A scheme that is kind to your congestion window can be unkind to your CPU. The point is not that one of these is right, only that the window question has a different answer for every one of them, and it is worth asking before the certificate arrives rather than after.

This generalises past nginx. Any QUIC implementation with a fixed initial congestion window and a rule about what may be sent once that window is exhausted has the same arithmetic to check, and the check is small: take your initial window, multiply by your reduction factor, and compare the result against the bytes a full server flight puts in flight. If the second number is larger, a single lost Initial packet is enough.

What #1617 became, on 4 September 2026

At 05:15 Riga time on 4 September the #1617 branch was force-pushed. The CUBIC line measured above is no longer there. The pull request keeps its number, and everything else about it changed: the title is now QUIC: let handshake recovery bypass congestion, the head is b941661a, and the diff touches four files for 14 added lines and 6 removed. It is rebased onto nginx 1.31.5 and its body says it closes #1616.

What it does is neither the CUBIC change nor ours, but it borrows one hunk from each side of that argument. ngx_quic_resend_frames() takes a new argument, and the output loop learns to honour the flag it sets:

/* ngx_event_quic_ack.c - ngx_quic_resend_frames() */
+ if (f->type == NGX_QUIC_FT_CRYPTO) {
+     f->ignore_congestion = ignore_congestion;
+ }

/* ngx_event_quic_output.c - ngx_quic_output_packet() */
- if (ack_only && f->type != NGX_QUIC_FT_ACK) {
+ if (ack_only && f->type != NGX_QUIC_FT_ACK
+     && !f->ignore_congestion)
+ {

/* ngx_event_quic_ssl.c - "speeding up handshake completion" */
- ngx_quic_resend_frames(c, ctx);
+ ngx_quic_resend_frames(c, ctx, 1);

/* ngx_event_quic_ack.c - ngx_quic_detect_lost() */
- ngx_quic_resend_frames(c, ctx);
+ ngx_quic_resend_frames(c, ctx, 0);

The second hunk is character for character the one we proposed on 30 July, and it is the one that matters: without it ignore_congestion is read only in ngx_quic_frame_sendto(), so a flag set anywhere else has no effect on the main output loop. Our own first attempt failed for exactly that reason, 11 failures in 200.

The difference is who is allowed to set the flag. Ours set it on every retransmitted CRYPTO frame below the application encryption level, whatever caused the retransmission. This one sets it only when ngx_quic_resend_frames() is called with a 1, and there is exactly one such call site: the branch in ngx_quic_handle_crypto_frame() that runs when the server receives a duplicate CRYPTO frame from the client - the "speeding up handshake completion" path the core developer named on 5 August. The ordinary loss-detection path passes 0, and because the assignment is unconditional, that call actively clears the flag again.

That scope makes the fix depend on the server receiving a duplicate CRYPTO frame at all. It does receive them - and the fix still never runs. Instrumenting b941661a with three extra log lines, one at the branch itself, one on the resend argument and one on any frame passing the acknowledgement-only gate, gives an unambiguous answer across 300 connections at 8% loss:

$ grep -o 'quic dupcrypto level:[0-9]* sent_empty:[0-9]*' debug.log | sort | uniq -c
     55 quic dupcrypto level:0 sent_empty:1

$ grep -c 'ic:1' debug.log
0
$ grep -c 'bypass ack_only' debug.log
0

The branch condition is satisfied 55 times, every one of them at the Initial level. Not once does the bypass argument reach the resend, and not one frame ever passes the acknowledgement-only gate. Both client behaviours from the previous section appear inside this run - four of the six deadlocks receive duplicates, 22 of them in one connection and 11 in each of the others, and two receive none - so this is not a case of the client happening not to repeat itself. The reason is eight lines inside the same branch, in a guard that predates all of this: the resend runs only if (!ngx_queue_empty(&ctx->sent)), and by the time the duplicate arrives that queue is empty. Loss detection got there first:

quic frame tx init:1 CRYPTO len:1124 off:0
quic frame tx init:2 CRYPTO len:54 off:1124
quic detect_lost pnum:1 thr:58 pthr:3 wait:0 level:0
quic resend packet pnum:1 level:0 ic:0        <- loss detection drains ctx->sent, and clears the flag
quic congestion lost t:1113418315 win:8312 if:11875
quic dupcrypto level:0 sent_empty:1 off:1521   <- the duplicate arrives to an empty queue
quic dupcrypto level:0 sent_empty:1 off:1521

ngx_quic_detect_lost() calls the same function with a 0, which requeues the frames, empties ctx->sent, and - because the new assignment is unconditional - sets ignore_congestion back to 0 on the CRYPTO frame it just requeued. The client's duplicate then arrives to a queue with nothing in it, the guard fails, and the code that would have set the flag is skipped. The output-loop hunk, which is the half that actually does the work, never runs at all on this chain. On the smaller chain it sometimes does, and that turns out not to save the handshake either.

Run as a failure-rate series rather than a trace, on the same window-filling chain, three builds off one base and the block order rotated between rounds, that comes out as:

Do not read 32 against 25 as a partial fix. It is tempting, and the per-block numbers are the reason not to: stock's four blocks run 3, 9, 10, 10 and the patched build's run 10, 6, 2, 7, which is a spread wide enough to produce that gap from nothing. The trace settles what the totals cannot: on this chain the bypass never executes once in 300 connections, so in this path the two binaries are the same binary. We are reporting the counts because we ran them, and the blocks beside them because a single block would have supported any conclusion you liked.

On the default chain the signal does fire, and it still deadlocks

Everything above is the 10.5 KB window-filling chain. Repeating both the trace and the series on the 3.9 KB chain the reproduction bundle ships by default gives a different answer to the first question and the same answer to the second. Same instrumented build, 8% loss, 40 attempts, 10 failures:

$ grep -o 'quic dupcrypto level:[0-9]* sent_empty:[0-9]*' pr-small-debug.log \
    | sort | uniq -c
      2 quic dupcrypto level:0 sent_empty:0
     53 quic dupcrypto level:0 sent_empty:1

$ grep -c 'ic:1' pr-small-debug.log
7
$ grep -c 'bypass ack_only' pr-small-debug.log
3

Two of the 55 duplicate arrivals found ctx->sent still populated - both on the same connection, where the client's duplicate beat loss detection to the queue. That connection requeued two Initial packets and all five outstanding Handshake packets with the bypass set - seven resends, which is the ic:1 count above - and three frames went out through the acknowledgement-only gate:

quic frame tx init:1 CRYPTO len:1124 off:0
quic frame tx init:2 CRYPTO len:54 off:1124
quic dupcrypto level:0 sent_empty:0 off:1521   <- the duplicate wins the race
quic resend packet pnum:1 level:0 ic:1
quic congestion lost t:1116889971 win:3602 if:5146
quic resend packet pnum:0 level:2 ic:1
quic resend packet pnum:1 level:2 ic:1
quic resend packet pnum:2 level:2 ic:1
quic resend packet pnum:3 level:2 ic:1
quic resend packet pnum:4 level:2 ic:1
quic dupcrypto level:0 sent_empty:0 off:1521
quic resend packet pnum:2 level:0 ic:1
quic dupcrypto level:0 sent_empty:1 off:1521   <- drained from here on
        ... nine more, every one sent_empty:1 ...
quic bypass ack_only ftype:6              <- CRYPTO out through the gate
quic bypass ack_only ftype:6
quic resend packet pnum:3 level:0 ic:0        <- loss detection, taking the flag back
quic bypass ack_only ftype:6

There are eight resend lines on that connection, not seven: the last is the ordinary loss-detection path arriving afterwards and setting ignore_congestion back to 0, taking the flag away again on the one connection where the design worked. The three frames that did pass the gate are ftype:6, which is CRYPTO, so this is the retransmission the deadlock needs actually reaching the wire - twice before that eighth resend and once after it. The one that follows the clearing resend is not a contradiction: the assignment sits in the loop that moves frames out of ctx->sent, so it only touches what that call requeues, and a frame already sitting in ctx->frames from an earlier requeue keeps the flag it was given.

No STREAM frame ever follows. The connection ends in CONNECTION_CLOSE after 23 seconds, where a neighbouring connection that completed on the same build shows four received STREAM frames. So the honest statement across both chains is narrower than the one we would have written from the first trace alone: the signal is unreachable on the window-filling chain, reachable but rare on the default chain, and in the one firing we have observed it did not prevent the deadlock. Not "the branch can never be entered", and not "the client never re-sends".

The default chain as a failure-rate series, same three builds on the same base, four rounds of 50 each with the build order rotated between rounds:

The no-loss row is the control: all three builds complete 200 of 200, so the rig does not manufacture failures. Every failure in every cell of both series is the same one - curl: (55) at the ten-second handshake deadline - with no second failure mode anywhere in 1,600 attempts.

One more thing about our own column rather than anyone else's. A build that fails zero times in 800 attempts across both chains is also exactly the shape of a measurement that has stopped measuring, so the series is bracketed from both ends. At no loss, all three builds complete 200 of 200 - the rig does not invent failures. At 30% loss on the window-filling chain, the same patched build fails 2 of 30 and stock fails 15 of 30 - it can still report one:

stock 1.31.5        15 / 30 failures
with our two hunks   2 / 30 failures

Those two runs are controls, not rates: one block each, no rotation, and 30% loss each way is not a network anyone operates. All they establish is that the instrument moves in both directions, which is what the eight cells of zeros between them needed. The zeros are real.

It is worth being exact about whose idea this is, because the result is a negative one and it is not aimed where it might look. The scoped signal was proposed by the nginx core developer who diagnosed the bug, in the 5 August comment on the issue: signal ngx_quic_resend_frames() that this is the special handshake-completion case, rather than granting a blanket exemption to all handshake CRYPTO the way we had. The pull request implements a maintainer's design. What we measured is that the call site it uses is drained by loss detection before the trigger arrives, on one chain always and on the other nearly always - a placement problem, and not an argument for the wider exemption we originally proposed.

The same comment anticipates the rest of it. It notes that unacknowledged CRYPTO used to ride the PTO probe until the 2021 commit replaced that with PING frames, points at RFC 9002 section 6.2.4, and ends: "either of that or all together needs to be implemented". Connection *258 is evidence for the second half of that sentence. The bypass fired, seven packets were requeued with it, three frames left through the gate, and the handshake still died - so on this evidence the scoped signal is not sufficient on its own.

None of the failure-rate tables above has been re-run against this head, and none of them has been relabelled to suggest otherwise. Those figures were measured on nginx 1.31.3 carrying the CUBIC change; b941661a is different code on a different base, and carrying a column across would be inventing data rather than reporting it.

Corrections

Three claims on this page were published wrong and are recorded here rather than repaired quietly. Each is stated as it appeared, then as it stands.

#1617 was attributed to nginx

Published, from 2 August to 4 September 2026: "nginx replied with a fix two days later", with #1617 introduced as the fix nginx proposed and the section verifying it headed "someone else's fix". Corrected: #1617 was opened by a contributor with no prior association with the nginx repository, on an account created in May 2026, and it has never carried a review from anyone. nginx maintainers did engage with the report - one reproduced the bug on 4 August and an nginx core developer diagnosed it on 5 August - but neither opened that pull request. The two facts were adjacent in the timeline and we merged them.

It was caught by checking author_association on the thread rather than reading the number beside the word nginx, which is what should have happened when the pull request first appeared.

One connection was published as all six

Published, from 30 August to 4 September 2026: "All six look identical", above a folded qlog timeline showing the 54-byte tail arriving alone and then nothing but PING probes to the deadline. Corrected: the six failures split two ways. Four receive the 1,124-byte head with an acknowledgement attached; two receive only the tail, with none. And the connection printed as the general case is one of the two, which re-sent its entire ClientHello at 1,041 ms - a CRYPTO-carrying probe that the folding had removed from the very timeline used to argue the client sends PINGs.

Both errors are the same error, and this page had already warned about it in the section above: the server-side captures were checked against each other and found to vary, then the client-side ones were described from a single file. Re-reading the archive with a script rather than an eye found it in about a minute.

A measurement column named a pull request, not a patch

Published: failure-rate and congestion-state tables with a column headed "With #1617". Corrected: those builds carried one specific change - ssthresh derived from w_prior rather than in_flight - which sat at that pull request between 1 August and 4 September 2026 and is no longer there. The numbers are unchanged and were correct as measured; the label was a pointer to a moving target, and the columns now name the code instead. A pull request number is not a stable identifier for a patch, and a table that uses one as a column heading goes wrong silently the moment the branch is force-pushed.

Reproduce it yourself

None of this has to be taken on trust. The bundle is public and it is the same one the maintainers were given: github.com/Ozy-666/nginx-quic-initial-repro. It needs root for namespaces and netem, an nginx built --with-http_v3_module against a TLS library offering X25519MLKEM768, and an HTTP/3-capable curl.

git clone https://github.com/Ozy-666/nginx-quic-initial-repro
cd nginx-quic-initial-repro

./make-certs.sh          # 3-cert RSA-4096 chain, ~3.9 KB DER
./setup-netns.sh         # srv+cli namespaces, MTU 1500, 20 ms + 2% loss
mkdir -p html logs tmp && echo ok > html/index.html
ip netns exec srv nginx -c "$PWD/nginx.conf.example" -p "$PWD"
./run.sh 150

Expect roughly 4 to 5% of attempts to hang until the client's handshake timeout. To confirm the diagnosis rather than just the symptom, set ssl_ecdh_curve X25519; and run it again: the ServerHello fits in one packet and 100 out of 100 succeed.

For the window-filling case, build the larger chain and point ssl_certificate at it:

./make-bigchain.sh pki-big 300     # ~10.5 KB DER

What we would tell ourselves at the start

Stop defending your own build. We lost hours to "it is probably our patch" because that was the humble assumption and humility felt like rigour. It is not. It is just another untested hypothesis, and the fact that it is unflattering does not make it more likely to be true.

Take the capture earlier. The logs said nothing for a day. The capture said almost everything in ten minutes, because QUIC Initial packets decrypt without a keylog and nobody remembers that until they need it.

A reproduction is worth more than an analysis. The analysis in our report may well have been ignored. What produced a fix in two days was a script that made the failure happen on someone else's machine.

Test the clever thought before you publish it. Two of ours were wrong, and they were the two that felt most like findings.

The general form of the bug is worth carrying away even if you never touch QUIC. Two subsystems were each behaving correctly on their own terms: one decided a frame must be retransmitted, the other decided nothing may be sent. Neither is wrong, neither logs an error, and the failure lives entirely in the assumption they do not share - that something in flight will eventually be acknowledged. Wherever a rate limit governs the only message that could lift the rate limit, the same deadlock is available.