The record · 6 September 2026
Which websites block AI crawlers?
58,098 of them name one and refuse it. 1,848 ask to be paid. Nearly everyone else has never opened the file.
Every domain in the record has its own page: what it declared, why we decided that, and the status its server answered with. 449,070 domains, swept on 6 September 2026, of which 347,935 answered.
The ones asking to be paid
Only 1,848 domains in the whole record publish a licence or return a price — 0.53% of everything that answered. These are the names you would recognise.
Well-known sites that shut the door
58,098 domains name an AI crawler in their rules and refuse it. Newspapers got there first; everyone else is catching up.
Household names that declare nothing
286,854 domains answered and said nothing at all, 82.4% of those that answered. Their pages are free to take, and mostly nobody chose that.
What the five words mean
The site names an AI crawler in its robots.txt and tells it not to read the site.
The site publishes a licence, or answered with a price or an HTTP 402.
A Content-Signal preference and nothing else. A stated view, not a rule.
The site answered and declared nothing, so AI crawlers are permitted by default.
The site never answered our request at all, so there is nothing to read.
Want the live version for your own site, rather than what we wrote down in September? Run the free site review. The method is at /data and the same record is available over an API.