Banned by Our Own Refusals.

What we wrote at the time that does not survive re-reading

The internal note written on the day states that no prior reputation was involved and that each address's first log line was its own ban. Re-reading the observation ledger for this article, that is not what it says. All three bans list recidivist - a feature computed from our own past bans - among the contributing signals, and one of the three addresses has an earlier record from 25 June 2026.

The correction does not weaken the finding. It sharpens it, because it means the chain had one more self-referential link than we credited at the time: a prior ban was helping to license the next one. That specific defect - a feature derived from our own enforcement history being spent as evidence - had already been fixed once, in a different place, on 31 July. It was still reachable here.

The test that separates their mistake from yours

The first instinct after an incident like this is to enumerate the refusals you know are benign and exclude them. We did that. It is the wrong shape, and it is precisely why the crawler ban happened: somebody had enumerated several benign sources of a 444, and nobody had enumerated the Range: rule, so it counted as evidence by default.

The question that actually separates the two populations is about determinism, not about intent:

A 403 stays on the client's side of the line even though it is our policy that produced it, because the code is deterministic given the request. Treating it as ours would over-correct and blind detectors that legitimately depend on it. That decision is arguable, and we have written down that it is arguable rather than pretending the boundary is clean.

400 we classify as the client's, conservatively and with one caveat from our own history: a hardening setting on this server manufactured 400-class refusals for valid clients for five weeks. Any configuration can turn a client-caused code into a self-caused one, which is an argument for testing what your edge actually emits rather than reasoning from the specification.

On an encrypted-DNS endpoint, every 4xx is yours

This is where the general problem becomes a specific one for anyone running DNS over HTTPS. A DoH endpoint is normally a single exact-match location serving one path. Work through what a client can actually receive there:

• 404 is impossible. An exact-match location either matches or is not entered; there is no missing sub-resource to ask for.
• 403 is normally remapped. A resolver that does not want to advertise its policy surface turns policy rejections into a hangup.
• 429 is the rate limiter. That is a statement about our capacity, not their conduct.
• 444 is every remaining rejection. Method rules, content-type rules, request-shape rules - all of them ours.

So on that endpoint the set of 4xx responses that constitute client evidence is empty. A detector gated on "4xx from a suspicious network range" has, on the DoH tap, no evidence source at all. It cannot distinguish a scanner from a browser, because the only thing it can observe is our own reflexes.

We had exactly such a detector. It ran from 28 June to 16 August 2026, and this is its complete recorded life, counted from the observation ledger:

The last figure is the one that ended the argument. Across all 67 fires, the contributor the observer recorded every single time was recidivist - our own enforcement history. Six of the 67 added exactly one more signal, and that signal is set under the same violation flag that counts the 444s we mint. Not one fire was attributed to a behaviour observed independently of something we had done.

Meanwhile the model watching those same decisions disagreed with almost all of them:

We rewrote the detector on 16 August at 17:29 to count only responses we had not minted. Fifty-three minutes later, at 18:22, we deleted it. Once the self-minted evidence was removed there was nothing left for it to fire on that a better-specified detector was not already catching, and a rule whose entire input was our own output does not deserve a second chance to be tuned.

Why it survived seven weeks: the false positive that cannot report itself

Forty-one of those 67 fires were one address - an external user of this resolver behind a hosting-provider VPN exit, not an operator device, not a test client. It kept getting banned, and we did not hear about it once.

Our fastest false-positive detector is the operator's own phone and laptop. When a rule breaks them, we find out in seconds, because the person who can change the rule is the person locked out. That mechanism works, and it is the reason we never add our own addresses to an allowlist - an allowlisted canary stops being a canary.

Its blind spot is structural and worth stating plainly: the false-positive rate you can see is the false-positive rate on the devices of people who are able to complain. A stranger whose DNS stops resolving does not open a ticket. They retry, they get a broken browser tab, and eventually they change resolver. Nothing in the system produces a signal. That detector ran for seven weeks at a false-positive rate we now consider unacceptable, and it survived precisely because it never once hit the one user who would have said so.

The remedy is not a better canary. It is to stop treating absence of complaint as evidence, and to read the population a rule actually acts on - the addresses, the distribution, whether one address dominates - before deciding the rule is working. Ours dominated at 61%.

The second loop: the evidence we destroyed

Manufacturing incriminating evidence is the obvious half. The other half took longer to see, because it produces no log line at all: enforcement that erases exculpatory evidence, and then reads the absence it created as innocence never earned.

This resolver grants a client standing based on a history of clean queries. That standing is what keeps an ordinary user out of a flood response when the traffic is heavy. On 18 August, during a sustained flood, one such client produced these four rows, which are all four rows it has:

Two requests per second apart, 69 seconds apart in time, and the outcome moved from 59 seconds to 72 hours. Nothing about the client changed. What changed is that the 59-second ban had, as part of its cleanup, deleted the row recording the client's earned history.

That deletion was not punitive and it was not a mistake in intent. A banned address must not retain a kernel-level fast path - that is correct and non-negotiable. The defect was that one row was doing two unrelated jobs: it was both the membership token for the fast path and the ledger of what the client had earned. Revoking the first meant destroying the second, and the destruction was invisible, because a missing record produces no log line, no counter and no complaint. It produces a slightly harsher decision, later, that looks entirely reasonable in isolation.

The escalation is the part that makes this a loop rather than a one-off unfairness. One false positive manufactures the conditions for a worse false positive. Left alone, that ratchets in one direction only.

Why no threshold could have separated these populations

The rate that triggered all of this was 14 to 16 requests per second. It is worth asking what that number is actually measuring, because we had been comparing it against a fixed bar chosen to separate humans from machines.

It is a five-second average. The counter is a five-slot ring of one-second buckets, and idle windows are evicted after thirty seconds. "16 rps" means sixteen requests in the last five seconds.

Now consider what a person does when they restore a browser session. Ten tabs reopen at once. Each one resolves its own hostname, and a modern browser asks for both an address record and an HTTPS record for each, so twenty lookups leave in a burst - and a burst is, definitionally, a five-second phenomenon.

The measurement window and the event being measured are the same size. When that is true, no value of the threshold can separate the two populations: raise it and you miss real floods, lower it and you catch tab restores, and there is no setting in between where humans sit on one side and machines on the other. The fix for a bad constant is not a better constant. It is a measurement whose window is not the same shape as the thing you are trying to exclude - a per-client baseline, a budget spent over time, a duration term - anything that can tell a five-second burst from a six-hour siege.

There is a second trap in the data here, and we walked into it before catching ourselves. Across 25,854 durable enforcement records on this resolver, only 11 carry a trust label indicating an established client. It is tempting to read those 11 rows as "what established clients look like". They are nothing of the kind: they are the distribution of established clients that a detector proposed acting against, which by construction is the tail. Choosing a threshold from that sample would be choosing it from the false positives.

What we changed

Four changes, in the order they matter. None of them is a threshold.

1. Label provenance instead of erasing it. Our first attempt at this rewrote forgiven refusals into a different status code before the log reader saw them. That works, and it is wrong: it destroys the distinction between a rewritten refusal and a genuine one, and it forces the decision onto the one layer that cannot see per-address history. The edge now labels each refusal with whose fact it is; the reader decides what to do about it. Labelling composes, erasure does not.

2. Make the default safe, not the exceptions enumerated. A refusal counts as ours unless a rule explicitly promotes it to being about the client. The inverse - enumerate the benign refusals, let everything else count - is what produced the crawler ban, and it fails silently every time somebody adds a new rule. There is exactly one promotion rule today, and it is the case where the refusal genuinely is a verdict about the client.

3. Separate revoking a privilege from erasing a history. The row that held both jobs now holds a revocation marker alongside the earned record. A ban withdraws the fast path immediately; a soft, heuristic ban charges the history rather than deleting it; a ban on hard evidence still clears it entirely. Replaying the four rows above against the new behaviour, the second ban lasts 59 seconds instead of 72 hours.

4. Make the omission fail the build. The reason this class of defect kept recurring is that the mistake is always an absence - a path that never asked, a release that never ran. Absences do not show up in review. They now fail compilation: a static check walks the source and refuses to build if an enforcement path neither consults the trust state before acting nor settles its obligations after. Writing that check found a fourth path nobody had noticed, on the code that restores state after a restart.

If you run Fail2Ban, CrowdSec, or your own log watcher

You almost certainly have some version of this. The stock pattern - watch the access log, count 4xx responses per address, ban above a threshold - is the exact shape of the bug, and it is only safe if none of the codes it counts are ones your own configuration produces. Three checks, in increasing order of effort.

Check one: list the status codes your config emits by hand. Every one of these lines is a verdict of yours, not an observation about a client:

# every status your own rules mint, with the file and line that mints it
grep -rnE 'return +(4[0-9]{2}|444)' /etc/nginx/
grep -rnE 'error_page +[0-9 ]+ *=?(4[0-9]{2})' /etc/nginx/
grep -rnE 'limit_(req|conn)_status' /etc/nginx/
grep -rn  'deny ' /etc/nginx/

# now the other side: what your banning layer counts as evidence
grep -rn 'failregex' /etc/fail2ban/filter.d/ | grep -E '4[0-9]{2}'

Any status appearing in both outputs is a loop. On a default nginx and a stock nginx-4xx-style jail there is usually at least one.

Check two: trigger your own refusal and watch the counter move. From an address you control and can afford to have banned, send whatever your config refuses - a blocked method, a blocked path, a header rule - and read the jail's counter afterwards. If the count went up, your policy is feeding your ban system.

# from a host you control - use a path or method YOUR config refuses
for i in $(seq 1 5); do curl -s -o /dev/null -w '%{http_code}\n' \
  -X TRACE https://example.com/; done

# did your own refusal become evidence against you?
fail2ban-client status nginx-4xx

Check three: ask what a ban does to your allowlists. If banning an address also clears any per-address state you keep - a trust score, a verified-client cache, a rate-limit exemption - then your first false positive is quietly setting up your second. Revoking access and forgetting history are different operations and they should be different lines of code.

A closing note on 404, because it is the exception that makes the rule usable. It is the one code in the set that is genuinely the client's doing: they asked for something that is not there. If you want a 4xx-based jail that means something, count 404s and nothing else - and be aware that this narrows it to path enumeration, which is a real attack but a different one from what most people think they are catching.

The general lesson

A defence that reads its own output as input will eventually convict on evidence it created. That sentence covers both halves of this article, and neither half needed a bug to happen: the edge refused a header exactly as configured, the log reader counted a 4xx exactly as configured, the ban cleaned up its state exactly as designed. Every component was correct. The composition was not.

There is a second, less comfortable lesson about how long this survived. Both loops were invisible for the same reason - they produce silence rather than errors. A self-inflicted ban looks like a successful detection. An erased history looks like a client that never earned anything. Neither writes a line saying something went wrong, which is why they were found by reading population statistics rather than by an alert firing.

If you take one operational habit from this: for every signal your enforcement consumes, be able to say where it came from. Not what it means - where it came from. A signal that traces back to your own configuration, your own load, or your own past decisions is not evidence, however confident the number attached to it looks. On our worst detector, that trace led back to us on all 67 occasions it fired, and we had never once followed it.

We publish these because the alternative - being corrected by someone else's crawler team, or by a user who simply left - is worse, and because we have had to correct our own published claims before. If something here contradicts what you can measure on your own stack, we would rather hear it.