TL;DR

  • Three verified Meta crawlers were hard-banned on 5 and 6 August while requesting /robots.txt. The evidence was a 444 our own edge rule had returned to them.
  • The same rule had been breaking link previews site-wide for as long as it existed. We found that out because it also banned the crawlers.
  • One detector fired 67 times in seven weeks across 16 addresses. On all 67 the only contributor recorded was our own past bans. It was rewritten on 16 August at 17:29 and deleted at 18:22.
  • The second loop runs backwards: a 59-second ban erased a real user's earned history, and 69 seconds later the same client drew 72 hours.
  • On a DoH endpoint every 4xx is self-minted - 404 is impossible on an exact-match location, 403 is remapped, the limiter answers 429.

The chain

A crawler sends a Range: header. An edge rule answered every such request by closing the connection - nginx's non-standard 444, which the server never emits on its own. The log reader downstream counted that 444 as a violation by the client, a header-poor HTTP/2 heuristic added its score, the total crossed its bar, and the ban fired.

The classifier's confidence was identical to three decimal places across all three addresses on three different days. That uniformity is what gave it away: three clients producing the same number is not three detections, it is one deterministic condition being met.

The test that separates the two

Not intent - determinism. Would an identical request from a well-behaved client receive this same status code, regardless of your current load and policy state? For 400, 403, 404 and 408 the answer is yes and the fact belongs to the client. For 429 and 444 it is no, and the fact is yours.

The first instinct is to enumerate the refusals you know are benign and exclude them. That is the wrong shape, and it is exactly why this happened: several benign sources of a 444 had been enumerated, and nobody had enumerated the Range: rule, so it counted as evidence by default. The default has to be safe.

The loop that runs backwards

On 18 August, during a sustained flood, a client with an earned history of clean queries drew a 59-second ban. That ban's cleanup deleted the row holding the history - correctly revoking a fast path, and destroying an earned record that happened to live in the same row. Sixty-nine seconds later the same address was a stranger, and a stranger gets a stranger's duration: 259,199 seconds.

The classifier watching both decisions returned shed, not ban, on each of them. The 72 hours came from the rule path.

What changed

The edge now labels each refusal with whose fact it is instead of rewriting it, so the layer that can see per-address history is the layer that decides. Revoking a privilege and erasing a history are now separate operations. And because this class of defect is always an absence - a path that never asked, a release that never ran - a static check now fails the build when an enforcement path does neither. Writing that check found a fourth path nobody had noticed.

Every number in the guide was re-read from this server's durable ban and observation ledgers on 23 August. The figures that did not reproduce were dropped rather than repeated, and one claim from the original incident note is corrected in the article.