Runbook Published: When a Public Blocklist Bans Googlebot

A crowd-sourced IP blocklist this service imports had listed real Googlebot addresses. The firewall dropped them before nginx ever saw a request, and the site began to fall out of search. The full incident writeup is now published as a guide, with every tool included.

This service imports public IP blocklists to keep abusive traffic away from the resolver. One of them, the crowd-sourced blocklist_de feed, carried entries inside Google's published crawler range. Imported verbatim twice a day, it told the firewall to drop Googlebot at layer 4, before nginx could log anything. The site stayed fully reachable for visitors while Search Console impressions bent toward zero.

What changed

Crawler-gated logging: nginx now writes a dedicated log for search-bot User-Agents only, so crawler traffic is visible without logging normal visitors.
Identity verification: a small script checks every logged crawler IP by forward-confirmed reverse DNS, the method the search operators themselves document.
Exempt, not whitelist: an nftables set, rebuilt automatically from nine official crawler IP feeds, cancels only the imported-blocklist drops. Rate limits and behavioural guards still apply to every packet, so a forged crawler address buys nothing.
Feed decommissioned: the offending crowd-sourced feed was disabled, as recorded on the server info page.

The full writeup

The complete account, from the flat Search Console graph to the recovery, is published as a runbook: When Your Blocklist Bans Googlebot. It includes the nginx configuration, the verification script, and the full exempt-set updater, ready to use on any service that imports blocklists.