The whole guide in one picture: a whitelist trusts an address; an exempt only cancels one drop.
Every public service that filters traffic eventually inherits someone else's mistake. Here is ours, in full, with the commands to check your own.
The symptom: a site that quietly disappears
It started with a flat line. The impressions graph in Google Search Console, normally a gentle hillscape, had bent toward zero and kept going. My first thought was a reporting glitch, so I opened the site - it loaded instantly. Phone on mobile data: fine. A different network: fine. Uptime clean, TLS valid, nothing in the error log. Every instrument I trusted said the site was healthy, and the one graph that mattered said it was disappearing.
So I tailed the nginx logs, expecting to catch Googlebot getting a 500 or a redirect loop. Instead I found nothing at all - no Googlebot, no errors, just the ordinary trickle of real visitors. That emptiness was the clue I nearly walked past. If the crawler were being served a bad page, nginx would have logged the hit; seeing no hit meant the request had never reached nginx. It was being dropped a layer below, at the packet filter, before the web server ever ran. The site was not broken. It was invisible, and only to the single visitor that decides whether anyone else ever finds it.
That combination is the tell. When a site works for you but not for Googlebot, the block is almost never in the application. It is lower down, at a layer that answers the crawler differently than it answers you - and the most common culprit is a packet filter that drops the request before the web server ever runs.
This service sits behind a firewall that imports public blocklists to shed abusive traffic. If Googlebot's address lands on one of those lists, the packet is dropped at layer 4. Nginx never logs it because nginx never sees it. From the crawler's side the server is simply unreachable, and after enough unreachable visits, Google stops trying. The site does not get penalised; it gets forgotten.
Why it happens: a blocklist is someone else's opinion
Public IP blocklists are enormously useful and completely unaccountable to you. They are aggregates of reports - a machine somewhere probed a honeypot, brute-forced an SSH port, or sent spam, and its IP was added. The problem is that addresses are reused and shared. A cloud range that hosted an abuser last week hosts something innocent this week. Large crawler fleets ride on wide provider ranges, and those ranges pick up collateral listings.
In our case the culprit was specific, and worth naming. We pull several public feeds through the FireHOL iplists.firehol.org catalogue, and one of them - blocklist_de, a crowd-sourced feed of addresses that reportedly attacked someone's SSH, mail or web service - carried entries sitting inside Google's published 66.249.0.0/16 crawler range. Bingbot and Yandex addresses were swept up the same way. Import that feed verbatim on a twice-daily job, and twice a day the firewall was told, in effect, to drop Googlebot along with the genuinely bad traffic. We had integrated the list blindly, trusting its verdicts wholesale.
The lesson is not "stop using blocklists." They do real work. The lesson is that importing a list is delegating a decision, and some decisions should not be delegated without a second opinion. A resolver operator already knows this instinct from the DNS side - it is why DNS ad-blocking lists need curation and false-positive handling rather than blind trust. The same discipline belongs at the firewall.
Two things had to change. First, we needed to see what crawlers actually received, so a problem like this could never again be invisible. Second, we needed a way to keep importing the lists while guaranteeing that a verified crawler is never among the casualties.
Step one: see what the crawlers actually get
This service keeps general access logging off for privacy - there is no per-request record of normal visitors. But a crawler is not a normal visitor, and to diagnose a crawler problem you have to be able to watch crawler traffic specifically. Nginx makes this a two-part change: classify the request by User-Agent, then log only the requests that match.
At the http level, a map sets a flag for known crawler User-Agents, a dedicated log format captures just the fields that matter, and an access_log with an if= gate writes only the matching hits:
# http { } block
map $http_user_agent $is_crawler {
default 0;
~*(Googlebot|GoogleOther|Google-InspectionTool|Storebot-Google|APIs-Google|AdsBot-Google) 1;
~*(bingbot|msnbot|BingPreview) 1;
~*(YandexBot|DuckDuckBot|Applebot|Baiduspider|SeznamBot|PetalBot|Sogou) 1;
}
log_format crawler '$time_iso8601 $remote_addr $host "$request" $status $body_bytes_sent "$http_user_agent"';
# general traffic stays unlogged; only crawler hits are written
access_log /var/log/nginx/crawlers.log crawler if=$is_crawler;One wiring detail matters. Nginx allows an access_log directive at the http, server and location levels, but a directive at a deeper level replaces the ones above it rather than adding to them. If your server block already sets its own access_log (many do), repeat the crawler line there and inside the location that serves your pages, so the crawler log is added at each level that would otherwise override it:
# repeat inside server { } and location / { } so it is not overridden
access_log /var/log/nginx/crawlers.log crawler if=$is_crawler;The result is a quiet log that stays empty except for search bots. Ours told the story immediately: over a window where Googlebot should have visited dozens of times, there were zero genuine Googlebot entries. The crawler was not being served pages badly; it was not arriving at all. That pointed the investigation straight past nginx and down to the firewall.
Step two: is it really Googlebot? The names don't lie
Before trusting any crawler, you have to answer a question that runs underneath this whole topic: how do you know a request claiming to be Googlebot really is? The User-Agent string is just text - anyone can send it. The source IP is more, but on connectionless transports it can be forged outright. Neither is proof on its own.
The answer the search operators themselves publish is forward-confirmed reverse DNS, and it turns on the one thing an impostor cannot fake: a name in a domain they do not control. It is two lookups. First, ask what name the IP reverse-resolves to, and check it sits in the operator's official domain. Then resolve that name forward and confirm it points back to the same IP:
# 1. reverse: what name does the IP claim?
$ dig +short -x 66.249.66.1
crawl-66-249-66-1.googlebot.com.
# 2. forward: does that name resolve back to the same IP?
$ dig +short crawl-66-249-66-1.googlebot.com
66.249.66.1 ← matches: genuineBoth halves are required. A reverse record alone proves nothing - anyone can point their own PTR at googlebot.com-looking text. It is the forward lookup, landing back on the same address, that closes the loop, because the forward zone for googlebot.com is controlled by Google. Each operator publishes its own suffix to check against:
| Crawler | Reverse-DNS suffix must end in |
|---|---|
| Googlebot | googlebot.com, google.com |
| Bingbot | search.msn.com |
| Applebot | applebot.apple.com |
| YandexBot | yandex.ru, yandex.net, yandex.com |
| Baiduspider | crawl.baidu.com, baidu.com |
| PetalBot | petalsearch.com, aspiegel.com |
| DuckDuckBot | no rDNS - check the published IP list |
We wrapped exactly this into a small script that reads the crawler log, does the two-step check for every unique IP, and prints GENUINE, FAKE, or a note for the handful of bots that publish a static IP list instead of usable rDNS. It is short, has no dependencies beyond dig and awk, and is yours to use in full:
#!/bin/bash
# check-crawlers.sh - verify that IPs in /var/log/nginx/crawlers.log are the
# search bots they claim to be, via forward-confirmed reverse DNS:
# 1. PTR lookup of the IP must land in the bot operator's official domain;
# 2. the A record of that PTR name must resolve back to the same IP.
# This is the verification method Google, Bing, Yandex and Apple document.
# UA-only matches (nginx $is_crawler map) can be faked; this script is the
# genuineness pass. DuckDuckBot has no stable rDNS - verify against
# https://duckduckgo.com/duckduckbot manually.
#
# Usage: check-crawlers.sh [logfile ...] (default: crawlers.log + rotated)
RE='\033[0;31m'; GR='\033[0;32m'; YL='\033[0;33m'; CY='\033[0;36m'; NC='\033[0m'
LOGS=("${@:-/var/log/nginx/crawlers.log}")
RESOLVER=127.0.0.1
# The server's own IPs (detected at runtime - survives IP changes). Curling the
# site from this box with a bot UA lands here as a UA-only "crawler"; that's an
# operator self-test, not a fake.
SELF_IPS=" $(hostname -I 2>/dev/null) 127.0.0.1 ::1 "
# claimed-bot -> expected PTR suffix(es), space-separated
expected_domains() {
case "$1" in
*[Gg]oogle*) echo "googlebot.com google.com" ;;
*bingbot*|*msnbot*|*[Bb]ing[Pp]review*) echo "search.msn.com" ;;
*[Yy]andex*) echo "yandex.ru yandex.net yandex.com" ;;
*[Aa]pplebot*) echo "applebot.apple.com" ;;
*[Bb]aiduspider*) echo "crawl.baidu.com baidu.com baidu.jp" ;;
*[Ss]eznam*) echo "seznam.cz" ;;
*[Pp]etal[Bb]ot*) echo "petalsearch.com aspiegel.com" ;;
*[Ss]ogou*) echo "sogou.com" ;;
*[Dd]uck[Dd]uck*) echo "LIST" ;; # published IP list, no rDNS
*) echo "" ;;
esac
}
printf "${CY}%-16s %-14s %-40s %s${NC}\n" "IP" "CLAIMS" "PTR" "VERDICT"
# unique IP + first UA token that identifies the bot
zcat -f "${LOGS[@]}" 2>/dev/null | awk '{
ip=$2; ua=""
if (match($0, /"[^"]*"$/)) ua=substr($0, RSTART+1, RLENGTH-2)
bot="?"
n=split("Googlebot GoogleOther Google-InspectionTool Storebot-Google APIs-Google AdsBot-Google bingbot msnbot BingPreview DuckDuckBot DuckDuckGo Yandex Baiduspider Applebot SeznamBot PetalBot Sogou", B, " ")
for (i=1; i<=n; i++) if (index(tolower(ua), tolower(B[i]))) { bot=B[i]; break }
if (ip != "" && bot != "?") print ip, bot
}' | sort -u | while read -r ip bot; do
case "$SELF_IPS" in *" $ip "*)
printf "%-16s %-14s %-40s ${YL}SELF (this server - operator test)${NC}\n" "$ip" "$bot" "-"
continue ;;
esac
doms=$(expected_domains "$bot")
if [ "$doms" = "LIST" ]; then
printf "%-16s %-14s %-40s ${YL}CHECK LIST (duckduckgo.com/duckduckbot)${NC}\n" "$ip" "$bot" "-"
continue
fi
ptr=$(dig +short +time=2 -x "$ip" @$RESOLVER 2>/dev/null | head -1)
ptr=${ptr%.}
if [ -z "$ptr" ]; then
printf "%-16s %-14s %-40s ${RE}FAKE (no PTR)${NC}\n" "$ip" "$bot" "-"
continue
fi
ok_dom=0
for d in $doms; do case "$ptr" in *.$d|$d) ok_dom=1 ;; esac; done
if [ "$ok_dom" = 0 ]; then
printf "%-16s %-14s %-40s ${RE}FAKE (PTR not in: %s)${NC}\n" "$ip" "$bot" "$ptr" "$doms"
continue
fi
fwd=$(dig +short +time=2 A "$ptr" @$RESOLVER 2>/dev/null)
if echo "$fwd" | grep -qx "$ip"; then
printf "%-16s %-14s %-40s ${GR}GENUINE (forward-confirmed)${NC}\n" "$ip" "$bot" "$ptr"
else
printf "%-16s %-14s %-40s ${RE}FAKE (PTR spoofed, A!=IP)${NC}\n" "$ip" "$bot" "$ptr"
fi
doneRunning it is two commands: look at what arrived, then verify it.
# 1. see the crawler hits that reached nginx
$ tail /var/log/nginx/crawlers.log
# 2. verify every IP in the log by forward-confirmed rDNS
$ bash /root/nginx-build/check-crawlers.sh
IP CLAIMS PTR VERDICT
66.249.66.1 Googlebot crawl-66-249-66-1.googlebot.com GENUINE (forward-confirmed)
40.77.167.1 bingbot msnbot-40-77-167-1.search.msn.com GENUINE (forward-confirmed)
45.9.20.5 Googlebot - FAKE (no PTR)That last row is the whole point: a request that claimed to be Googlebot, with no reverse record to back it. Running the checker against our own log is how we confirmed the real problem - not fakes getting in, but genuine Googlebot never arriving at all - and it is the same method that tells a real crawler from an attacker wearing its name. The IP can lie. The name, forward-confirmed, cannot.
Step three: the fix, and why it is not a whitelist
The obvious reflex is to whitelist the crawler ranges - tell the firewall to accept them unconditionally. We deliberately did not, and the reason is the heart of this guide.
A whitelist entry is a full accept. It does not just skip the imported blocklist; it skips everything - rate limits, flood fuses, behavioural bans, the lot. That would be safe only if a source IP were proof of who sent the packet, and it is not. On UDP and QUIC - which is exactly how encrypted DNS transports like DoQ and DoH/3 run - the source address is trivially forged. Whitelist Googlebot's ranges and you have published a recipe: stamp a Googlebot source IP on a flood, and it sails past every protection you have. You would be trading a search problem for a denial-of-service hole.
So instead of a whitelist, we use an exempt set that cancels exactly one thing: the drops that come from imported blocklists. Nothing else. Concretely, the drop rules for imported lists gained a single condition - drop if the source is on an imported list and not in the exempt set:
# illustrative nftables shape - drop imported-list traffic
# UNLESS the source is a verified crawler
ip saddr @imported_blocklist ip saddr != @crawler_exempt dropRead that rule back and the guarantee falls out of the logic itself: a rule that drops "listed and not exempt" simply cannot drop an address that is in the exempt set. A crawler IP that a blocklist got wrong is overridden at the moment the packet is evaluated - it is not edited out of the list, the list stays intact for everyone else. And crucially, the exempt set is referenced only by these imported-list rules. Every other defence - the rate meters, the flood fuses, the behavioural bans that ban an IP for what it actually does - runs before and after, untouched, applying to a forged-crawler packet exactly as it would to any other. That is the whole difference the diagram at the top draws: a whitelist trusts the address; an exempt cancels one specific drop and leaves the real defences in place.
This is the packet-layer mirror of the rDNS check. At the log layer, forward-confirmed DNS tells a real crawler from a faker by its name. At the packet layer, the exempt set refuses to let a raw IP - which might be forged - buy anything more than an exemption from lists that should never have contained it.
Keeping the exempt set honest, automatically
An exempt set is only as good as its contents. Crawler ranges change, so a hand-maintained list would rot into the same stale, wrong state we were trying to escape. The one authority on where a crawler operates is the operator itself, and most of them now publish machine-readable IP ranges for exactly this purpose. So the set builds itself from the source of truth:
- Collect from official feeds. A small updater pulls the published JSON range lists - Googlebot and Google's special-crawlers feed (which also covers Google-Extended for Gemini), Bingbot, OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Applebot, and Perplexity's two feeds - each in the standard .prefixes[].ipv4Prefix shape.
- Add a static file for the rest. A few legitimate crawlers publish no clean feed - Anthropic's ClaudeBot on its stable range, Yandex, PetalBot, SeznamBot. Those live in one small reviewed file, each range confirmed by the rDNS method above before it goes in.
- Validate before trusting. Every fetched prefix is checked for a sane CIDR shape, and anything broader than a /16 is rejected outright - a corrupted or hostile feed cannot smuggle in a huge range that would exempt half the internet.
- Fail safe, never empty. If a feed is unreachable, the last known-good copy for that source is kept. If, after everything, the merged result would be empty, the update aborts and leaves the live set untouched rather than flushing it.
- Apply atomically. The final list is written as a single flush-and-replace so the set is never half-populated for even an instant, and it runs on boot and on every blocklist refresh cycle, so the exempt set is always at least as current as the blocklists it protects against.
That is the design; here is the actual implementation, published in full. One thing to understand before running it: the script only ever refreshes the contents of the set, so three things have to exist around it - the set must be declared, your imported-blocklist drop rules must reference it, and the file it writes has to be included in your ruleset so the set survives a full reload. It reads its feed list and static ranges from two small files.
First, the feed registry, one name|url per line:
# name|url - official machine-readable crawler IP feeds (schema: .prefixes[].ipv4Prefix)
googlebot|https://developers.google.com/static/search/apis/ipranges/googlebot.json
google_special|https://developers.google.com/static/search/apis/ipranges/special-crawlers.json
bingbot|https://www.bing.com/toolbox/bingbot.json
openai_gptbot|https://openai.com/gptbot.json
openai_searchbot|https://openai.com/searchbot.json
openai_chatgpt_user|https://openai.com/chatgpt-user.json
applebot|https://search.developer.apple.com/applebot.json
perplexitybot|https://www.perplexity.ai/perplexitybot.json
perplexity_user|https://www.perplexity.ai/perplexity-user.jsonSecond, a short hand-reviewed file for the legitimate crawlers that publish no clean JSON feed - each range confirmed by the rDNS method above before it goes in:
# Trusted crawler ranges with no clean JSON feed (rDNS-verified via check-crawlers.sh).
# Anthropic ClaudeBot / Claude web-fetch (official stable /21)
160.79.104.0/21
# Yandex crawlers (from yandex.ru/ips; verified by rDNS)
213.180.203.0/24
5.255.253.0/24
45.148.64.0/22
# PetalBot (Huawei)
114.119.128.0/19
# SeznamBot
77.75.72.0/21The set and its wiring, declared once:
# one-time: declare the table and the interval set that holds the ranges
nft add table ip protection
nft add set ip protection crawler_exempt { type ipv4_addr\; flags interval\; }
# reference the set in your imported-blocklist drop rules (drop if listed AND not exempt):
# ip saddr @imported_blocklist ip saddr != @crawler_exempt drop
# and include the file the script writes, so a full reload restores the set:
# (in /etc/nftables.conf)
include "/etc/nftables-crawler-exempt.conf"And the script itself, which validates every fetched prefix, keeps a per-source last-known-good cache, falls back rather than emptying the set, and applies the result atomically:
#!/usr/bin/env bash
# Refresh the nftables crawler_exempt set from official crawler IP feeds.
set -uo pipefail
FAMILY=ip
TABLE=protection
SET=crawler_exempt
SOURCES=/etc/dns-bot-guard/crawler-sources.txt
STATIC=/etc/dns-bot-guard/crawler-static.cidr
CACHE=/etc/dns-bot-guard/cache/crawler-exempt
INCLUDE=/etc/nftables-crawler-exempt.conf
MIN_MASK=16
LOG=/var/log/crawler-exempt.log
log(){ printf '%s %s\n' "$(date -Is)" "$*" >> "$LOG" 2>/dev/null; }
_octets_ok(){ local IFS=.; local -a o=($1); [ "${#o[@]}" -eq 4 ] || return 1
for x in "${o[@]}"; do [[ $x =~ ^[0-9]+$ ]] && (( x>=0 && x<=255 )) || return 1; done; }
# format + octets + mask 1..32 (used for static, trusted)
valid_cidr_static(){ local c=$1
[[ $c =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}/([0-9]|[12][0-9]|3[0-2])$ ]] || return 1
_octets_ok "${c%/*}"; }
# same, plus reject mask broader than MIN_MASK (used for auto-fetched)
valid_cidr_auto(){ local c=$1 mask
valid_cidr_static "$c" || return 1
mask=${c#*/}; (( mask >= MIN_MASK )); }
fetch_source(){ local name=$1 url=$2 tmp out n=0
tmp=$(mktemp); out=$(mktemp)
if curl -fsS --connect-timeout 10 --max-time 30 --retry 2 -L "$url" -o "$tmp" 2>/dev/null; then
while read -r cidr; do valid_cidr_auto "$cidr" && { echo "$cidr" >> "$out"; ((n++)); }; done \
< <(jq -r '.prefixes[]?.ipv4Prefix // empty' "$tmp" 2>/dev/null)
if (( n > 0 )); then sort -u "$out" > "$CACHE/$name.cidr"; log "OK $name: $n prefixes"
else log "WARN $name: 0 valid prefixes, keeping cache"; fi
else log "WARN $name: fetch failed, keeping cache"; fi
rm -f "$tmp" "$out"
}
build_desired(){ local desired=$1
: > "$desired"
while IFS='|' read -r name url; do
[[ $name =~ ^[[:space:]]*# || -z ${name// } ]] && continue
fetch_source "${name// }" "${url// }"
done < "$SOURCES"
# Apple fallback: if applebot fetch never succeeded, seed 17/8 (Apple owns all of 17/8)
[ -s "$CACHE/applebot.cidr" ] || echo "17.0.0.0/8" > "$CACHE/applebot.cidr"
cat "$CACHE"/*.cidr 2>/dev/null >> "$desired"
while read -r c; do valid_cidr_static "$c" && echo "$c" >> "$desired"; done \
< <(grep -vE '^\s*(#|$)' "$STATIC" 2>/dev/null)
sort -u "$desired" -o "$desired"
}
main(){
mkdir -p "$CACHE"
local desired; desired=$(mktemp)
build_desired "$desired"
local count; count=$(wc -l < "$desired")
if [ "${1:-}" = "--dry-run" ]; then cat "$desired"; rm -f "$desired"; return 0; fi
if (( count == 0 )); then log "ERROR desired list empty, set unchanged"; rm -f "$desired"; return 1; fi
local elems; elems=$(paste -sd, "$desired")
{ echo "flush set $FAMILY $TABLE $SET"
echo "add element $FAMILY $TABLE $SET { $elems }"
} > "${INCLUDE}.tmp"
if nft -f "${INCLUDE}.tmp"; then
mv "${INCLUDE}.tmp" "$INCLUDE"
log "APPLIED $count prefixes"
else
rm -f "${INCLUDE}.tmp"; log "ERROR nft apply failed, set unchanged"; rm -f "$desired"; return 1
fi
rm -f "$desired"
}
# Only run main when executed, not when sourced (lets tests load functions)
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then main "$@"; fiRun it on boot and on every blocklist refresh - a small systemd service alongside your existing blocklist timer - so the exempt set is never staler than the lists it guards against. A --dry-run flag prints the merged list without touching nftables, which is the safe way to inspect what it would apply.
That closes the loop the original incident opened. Blocklists still get imported on their schedule and still drop the traffic they are meant to. But a verified crawler can no longer be collateral damage, because the same schedule refreshes an exempt set, built from the crawlers' own published ranges, that cancels precisely the drops that would have hit them - and nothing more.
Getting back into the index
Unblocking the crawler stops the damage; it does not undo it. Google re-crawls on its own schedule, not the moment you fix the firewall, so dropped pages return gradually. These steps, in order, speed the recovery and each one doubles as a check that the block is really gone:
- Confirm the crawler can reach you again. Turn on the crawler-gated log described above and run the verifier. A healthy log shows Googlebot arriving and passing the forward-confirmed rDNS check. Until it is getting through, nothing else on this list will help.
- Open Google Search Console for your site. Go to search.google.com/search-console. If you have not verified ownership yet, do that first - the DNS TXT-record method or the HTML-file upload - because the recovery tools live inside the verified property.
- Inspect a key URL and test it live. Paste an important page into the URL Inspection bar at the top, then click Test Live URL. That live fetch comes from Googlebot's own 66.249.x range, so a passing result is also end-to-end proof your firewall no longer drops the real crawler. If the live test cannot fetch the page, the block is not fully cleared - fix that before continuing.
- Request indexing. On a page that passed the live test, click Request Indexing to push it back into the crawl queue. This is rate-limited, so spend it on your most important pages - the home page and main landing pages - rather than trying to submit the whole site by hand.
- Resubmit your sitemap. In the Sitemaps report, submit /sitemap.xml again. This makes Google re-read the full list of URLs and re-queue them, instead of waiting to rediscover each page on its own.
- Monitor, and give it time. Watch the Pages (index coverage) report and the Performance impressions over the following days. There is no button that makes reindexing instant; Google works through the backlog at its own pace. The firewall fix is what actually matters - the rest is patience.
How long that patience lasts depends on how much was dropped and how large the site is. In our case it was fast: the Search Console graphs looked normal again within about two days of clearing the block, which honestly surprised me - I had braced for weeks. A larger site or a longer outage can take considerably longer, but a clean, verifiable fix is what makes recovery possible at all, and a quick one likely.
A steady climb back in the coverage report is what closes the incident: the same pages that quietly disappeared returning under their own crawl, now that nothing at the firewall is standing in the way.
Check your own service
If you run anything that imports blocklists - a firewall, a WAF, a resolver with an IP filter - three checks tell you whether you have this problem. First, are genuine crawlers reaching you at all? Turn on a crawler-gated log like the one above and watch it for a day; a healthy site shows regular Googlebot and Bingbot entries.
Second, verify any address you are unsure about with the two-step lookup - it works from any machine, no access to the server required:
$ ip=66.249.66.1
$ ptr=$(dig +short -x $ip); echo "$ip -> $ptr"
$ dig +short "${ptr%.}" | grep -qx "$ip" && echo GENUINE || echo "FAKE or misconfigured"
GENUINEThird, if you import a public list - a FireHOL feed like blocklist_de, or any other - ask it the question we should have asked first: does it contain any address inside a crawler's published range? Pull the operator's own IP feed and test the list against it before you trust its verdicts wholesale. That single check would have caught ours.
The broader habit is the takeaway. A blocklist is a useful opinion, not a verdict. Log what your filters do to the traffic you actually want, verify identity by the one signal that cannot be forged, and when you make an exception, make the narrowest one that solves the problem - an exemption from a specific drop, never a blanket pass.
The rest of the defences
Blocklists are one layer. What actually keeps a public resolver standing is the set of guards that never trust a raw IP in the first place.
How this service is runRelated reading: how DNS blocklists work and why they need curation, why a source IP can be forged, and building a hardened resolver end to end.
Ozy-666 builds and operates dnsdoh.art, an encrypted DNS resolver serving DoH, DoH3, DoT and DoQ. He maintains its Edge-optimised AdGuardHome and dnscrypt-proxy forks and the custom nftables/XDP filtering described in this guide. This account is first-hand: the incident, the diagnosis, and the fix all happened on that live infrastructure.