When Hardening Refused Every POST.

What it looked like

Nothing. That is the honest answer, and it is the reason this article exists. The site loaded. Browsers were happy. Monitoring was green. The resolver answered DoH queries over HTTP/3, DoT and DoQ without complaint, and answered HTTP/2 GET requests without complaint too.

What failed was a single combination: a request carrying a body, over HTTP/2. Put four probes side by side and only one of them breaks.

Client side, the failure announces itself as a retry rather than an error:

* upload completely sent off: 200 bytes
* HTTP/2 stream 1 refused by server, try again on a new connection
* REFUSED_STREAM, retrying a fresh connect
* Connection died, tried 5 times before giving up
* closing connection #5

Read that carefully. The body uploads successfully. The refusal arrives afterwards, and curl's reaction is not to report an error but to try again - five times, on five fresh connections, before it gives up. A client with a retry budget larger than curl's default never gives up at all.

HTTP/3 deserves a word, because "different module" undersells why it is safe. QUIC carries flow control in the transport itself, and a server states its per-stream receive limit in the transport parameters exchanged during the TLS handshake - before the client is able to send any application data at all. There is therefore no window in which a client can be entitled to assume a larger limit than the one in force, which is the entire mechanism of this bug. In nginx the equivalent knob is http3_stream_buffer_size, which sets the initial_max_stream_data_bidi_remote parameter governing the client-initiated streams a request arrives on. Lowering it does not produce this failure, because there is no pre-acknowledgement gap to fall into.

Why we set it to 16k

The temptation is to write this section as a confession. It is more useful as an argument, because the reasoning was not careless - it is the reasoning any anti-DDoS checklist will lead you to, and that is precisely what makes the directive dangerous.

Under an application-layer flood, the resource that runs out first is usually memory, and the number that governs it is per-connection cost multiplied by connections you cannot refuse fast enough. So you go through your config looking for per-connection allocations and you make them smaller. That is not folk wisdom; it is the correct instinct, and it is what every hardening guide tells you to do.

nginx allocates the preread buffer in full, per stream that carries a body:

buf = ngx_create_temp_buf(r->pool, h2scf->preread_size);

So the arithmetic is immediate and it is not wrong. With concurrency capped at 16 streams, the default 64 KiB buffer is up to 1 MB per connection. At 16k it is 256 KB. A four-fold reduction in the worst-case memory an attacker can make you allocate, for a buffer that - so the reasoning goes - only ever needs to hold a DNS query a few hundred bytes long. Free money.

And here is the part that makes it a trap rather than a blunder: every neighbouring directive rewards you for exactly this move. Lower client_body_buffer_size and you spool to disk instead of RAM. Lower client_max_body_size and oversized uploads are rejected early. Lower http2_max_concurrent_streams and you blunt Rapid Reset. Lower limit_conn and you cap the swarm. Each one degrades gracefully, or refuses something you wanted refused. They teach you that smaller is safer.

http2_body_preread_size looks like a member of that family, sits among them in every configuration, and behaves like none of them. Below one specific value it does not degrade, does not warn, and does not refuse selectively - it refuses everything with a body. There is no partial credit and no signal that you have crossed a line.

The code path

The refusal is four lines in ngx_http_v2_state_headers(), and it is worth reading in full because every clause matters:

if (!h2c->settings_ack
    && !(h2c->state.flags & NGX_HTTP_V2_END_STREAM_FLAG)
    && h2scf->preread_size < NGX_HTTP_V2_DEFAULT_WINDOW)
{
    ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
                  "client sent stream with data "
                  "before settings were acknowledged");

    status = NGX_HTTP_V2_REFUSED_STREAM;
    goto rst_stream;
}

Three conditions, all required:

1. The client has not acknowledged our SETTINGS frame. A latency-sensitive client writes its first request immediately after the connection preface rather than idling for a round trip. For the first request on a connection this is effectively always true - and a DoH client opening a connection to ask one question is as latency-sensitive as clients get.
2. The HEADERS frame does not carry END_STREAM. That flag means "this request is complete". A GET sets it; a request with a body does not. This single clause is the whole reason GET traffic was unaffected and the site looked healthy.
3. preread_size < 65535. True only if you lowered it. NGX_HTTP_V2_DEFAULT_WINDOW is 65535; nginx's compiled-in default for the directive is 65536. One byte of margin is the only thing standing between a stock nginx and this behaviour.

The comparison is a hard <, so there is no gradient to tune along. We tested the boundary directly on a throwaway server, changing nothing but the number:

preread=65534  POST/h2 -> 000
preread=65535  POST/h2 -> 200

One byte is the difference between a working server and one that refuses every upload it is offered.

Why nginx does this, and why it is not a bug

The answer is that the directive does a second job nobody reading its name would guess. Further down the same file, nginx writes its SETTINGS frame:

buf->last = ngx_http_v2_write_uint16(buf->last,
                     NGX_HTTP_V2_INIT_WINDOW_SIZE_SETTING);
buf->last = ngx_http_v2_write_uint32(buf->last, h2scf->preread_size);

http2_body_preread_size is also the HTTP/2 INITIAL_WINDOW_SIZE that nginx advertises, and the per-stream receive window it enforces. It is not merely a buffer size; it is a protocol commitment.

That makes the refusal correct. Until the client acknowledges our SETTINGS frame, it has not seen our smaller number, and RFC 9113 entitles it to assume the default window of 65535 bytes. A conforming, well-behaved, entirely innocent client may therefore send 65535 bytes of body into a buffer we sized at 16384. nginx cannot absorb that. Its options are to overflow, to buy unbounded memory at exactly the moment an operator asked it not to, or to refuse the stream. It refuses, and it says so in the log.

At 65535 and above the condition becomes unreachable: our advertised window is never smaller than what a client assumes before the acknowledgement, so the overflow it guards against cannot happen.

This was our misconfiguration, not an nginx defect. Though one detail is worth noting for anyone who thinks a validator should have caught it: nginx bounds this directive only from above, rejecting values greater than NGX_HTTP_V2_MAX_WINDOW. There is no lower-bound check, no warning at startup, and no note in the directive's documentation that a floor exists. The value 16k is, as far as every tool in the chain is concerned, perfectly valid.

Five reasons it stayed invisible

Any one of these would slow you down. Together they make the failure genuinely undetectable by normal means, and each one generalizes well past nginx.

1. The log line is INFO. A production error_log ... error; discards it before it reaches disk. The message exists, is accurate, names the exact problem - and is thrown away by a log level almost everyone runs.
2. GET works. Every human check - open the site, click around, run a page-speed test - exercises the path that was never broken.
3. HTTP/1.1 and HTTP/3 work. Clients that fall back succeed. A client with fallback logic fails on h2, drops to h3, gets its answer, and reports nothing to anyone.
4. nginx -t passes. The configuration is syntactically valid and semantically catastrophic. nginx -t has no opinion about behaviour, and we treated it as though it did.
5. The error is retryable by design. RFC 9113 defines REFUSED_STREAM to mean "this stream was not processed; retrying is safe". Clients obey. They do not surface an error to the user, they do not log a failure - they open a new connection and try again, into the identical wall. The protocol's own politeness is what buried this.

How long was it broken? We cannot tell you

This is the part of the post-mortem we would prefer to skip, so it gets its own section.

The fix went in on 26 July 2026. The oldest surviving artifact that still contains the bad value is a configuration backup dated 23 June 2026. That is five weeks we can prove. Anything before it is unknowable: older backups have rotated away, and the refusals never reached a log file, so there is no record to consult. We do not know whether it was five weeks or five months.

An earlier internal note put it at "three to four months". We removed that figure, here and in the public repository, because we could not support it. It was inference presented as measurement, which in a post-mortem is the specific failure mode worth guarding against - the outage was already invisible in the present; there is no need to invent visibility into its past.

The lesson is not about record-keeping. It is that a fault which produces no log produces no history, and a fault with no history cannot be scoped after the fact. By the time you find it, the question "how long?" may simply have no answer left in the system.

Reproduce it in 60 seconds

You do not have to take any of this on trust, and you should not test it against production. Everything below runs on localhost against a throwaway nginx, and the whole thing is three commands.

A self-signed certificate and a minimal server:

mkdir -p /tmp/h2repro && cd /tmp/h2repro
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes \
  -keyout key.pem -out cert.pem -days 2 -subj '/CN=example.com' \
  -addext 'subjectAltName=DNS:example.com,IP:127.0.0.1'
head -c 200 /dev/urandom > body.bin

Save this as /tmp/h2repro/h2.conf. It is the entire server - no includes, no site config:

daemon off;
worker_processes 1;
error_log /tmp/h2repro/error.log info;
pid /tmp/h2repro/nginx.pid;
events { worker_connections 64; }
http {
    access_log off;
    client_body_temp_path /tmp/h2repro;

    http2_body_preread_size 16k;      # <-- the trap

    server {
        listen 8443 ssl;
        http2 on;
        server_name example.com;
        ssl_certificate     /tmp/h2repro/cert.pem;
        ssl_certificate_key /tmp/h2repro/key.pem;
        location / { return 200 "ok\n"; }
    }
}

Start it and run the four probes. Note that the first command is the one that proves nginx -t is no defence:

nginx -t -c /tmp/h2repro/h2.conf     # passes. it always passes.
nginx -c /tmp/h2repro/h2.conf &

R="--resolve example.com:8443:127.0.0.1 --cacert /tmp/h2repro/cert.pem"
W='%{http_code} %{http_version}\n'

# 1. GET over h2          2. POST over h1.1
# 3. POST over h2         4. POST over h2 to a path that does not exist
curl -s -o /dev/null -w "$W" $R https://example.com:8443/
curl -s -o /dev/null -w "$W" $R --http1.1 -X POST --data-binary @body.bin https://example.com:8443/
curl -s -o /dev/null -w "$W" $R -X POST --data-binary @body.bin https://example.com:8443/
curl -s -o /dev/null -w "$W" $R -X POST --data-binary @body.bin https://example.com:8443/no/such/path

What you get, before and after changing that one value to 65535 and reloading:

And because the throwaway server logs at info, the explanation is sitting in the log the whole time - the same line your production server was discarding:

[info] client sent stream with data before settings were
acknowledged while processing HTTP/2 connection, client: 127.0.0.1

Clean up with nginx -s quit -c /tmp/h2repro/h2.conf.

Checking a live server

If you want to know whether this is happening to you right now, the reliable method is to stop discarding the evidence. nginx allows more than one error_log, so you can add a second one at INFO to its own file without touching the one you already have and without changing any behaviour:

error_log /var/log/nginx/error.log error;      # keep your existing one
error_log /var/log/nginx/h2debug.log info;     # TEMPORARY - remove after

Reload, wait a few minutes, then count what you find and who it is happening to:

# how many refusals
grep -c 'before settings were acknowledged' /var/log/nginx/h2debug.log

# which clients are affected, most-hit first
grep 'before settings were acknowledged' /var/log/nginx/h2debug.log \
  | grep -oP 'client: \K[0-9a-f.:]+' | sort | uniq -c | sort -rn

What it cost

Eight minutes of INFO logging on the live resolver, immediately before the fix:

Roughly 1,590 refusals an hour, and the retries never succeed - a fresh connection meets the identical pre-acknowledgement wall. The distribution is worth noting too: of ten affected addresses, one accounted for 115 refusals and the next for 36. Retry behaviour is not uniform, so a client with an aggressive backoff policy generates most of the noise while the quieter ones are invisible in the aggregate.

Those ten addresses were not attackers. They were people using the resolver, whose clients had been quietly hammering it for weeks because we told them, in the most standards-compliant way possible, that trying again was the right thing to do.

That inversion is the part worth sitting with. The setting existed to reduce load under flood. What it produced was flood-shaped traffic from legitimate users: instead of one request each, they generated a continuous stream of connection attempts. It is also worth knowing, if you find this on your own server, that a retry storm of this shape can trip generic rate limiting - so the same misconfiguration that breaks your clients can also get them throttled for the symptom.

What raising it actually costs

The original concern - memory - was real, so it deserves a real measurement rather than reassurance. Resident set size of the nginx worker, sampled at both values on the live server:

About +2.5 MB, roughly 3%, with no upward drift over 13 minutes - a steady-state shift, not a leak. That is the entire price of the four-fold saving we thought we were making.

The worst case is still worth understanding, because it is the number the original reasoning was aimed at. The buffer is allocated in full per body-carrying stream, so the ceiling is http2_max_concurrent_streams × 64 KiB per connection, bounded further by limit_conn. One detail catches people out: a small client_max_body_size does not shrink this allocation. The buffer is sized from the directive, not from what you are willing to accept, so capping bodies at 4 KB does not get you a 4 KB buffer.

What to do instead

Keep http2_body_preread_size at 65535 or above. nginx's default of 65536 is correct and there is no good reason to move it. Because the comparison is a hard <, there is no middle ground to explore: any smaller value refuses every body-carrying HTTP/2 stream, so tuning it downward is not a trade-off, it is an outage.

If bounding HTTP/2 memory is a genuine requirement, these are the directives that do it without breaking requests:

• http2_max_concurrent_streams - caps streams per connection, and is the real defence against Rapid Reset (CVE-2023-44487).
• client_max_body_size - rejects oversized requests early, before they cost you anything.
• client_body_buffer_size - controls what is held in memory versus spooled to disk.
• limit_conn - caps concurrent connections per key, which is what actually bounds the multiplication.

All four are in the annotated anti-ddos.conf in our nginx build repo, which now also carries this directive with a warning above it, so the next person reading that file for hardening ideas does not repeat the experiment.

The general lesson

A hardening change that pushes a limit below a protocol default stops being a capacity control and becomes an availability failure. That sentence is the whole article, and the specific directive is almost incidental - the same shape exists in any stack where a tunable doubles as a protocol commitment.

The habit that would have caught it is small: after tightening any limit, verify the thing you tightened still works. Not that the config parses, not that the site loads - that the specific operation the limit governs still completes. One curl -X POST over HTTP/2, run once after the change, would have saved five weeks. We had run nginx -t, seen it pass, and treated that as verification. It validates syntax. It has never had an opinion about behaviour.

And the second habit, which is harder: absence of a log line is not absence of a fault. Ours was logging faithfully the entire time, at a level we had chosen not to read, about an error the protocol defines as safe to ignore. Silence from a system is only evidence when you know the system would have spoken.