Questions people
ask first.

A program run by an AI company that visits your website and reads your pages, the same way Google's does. Some of them are collecting material to train a model. Others are fetching your page right now because somebody asked an assistant a question your site can answer.

They are not hackers and they are not doing anything unusual. They are just reading, at scale, and what they are allowed to read is up to you.

Through a small text file called robots.txt that sits at the root of your site. Almost every website has one. Yours was probably written by whoever built the site, or generated automatically by the platform it runs on, and has not been looked at since.

That is the honest situation for most businesses: a file you did not write is answering a question you did not know was being asked. The free check just reads it back to you.

For most small sites, the useful answer is: block the ones that only train, stay open to the ones that cite you.

A crawler that trains on your material takes your photographs, your descriptions and your prices, and sends you nothing back. A crawler that reads you to answer somebody's question names you in the answer, which is a customer arriving for free. Blocking everything is the common mistake, because it is the easy thing to do and it quietly removes you from a channel that is growing.

No, and we would rather say so than sell you the fantasy. A site with a million pageviews a month earns roughly $100 to $200 a month from crawler tolls at current rates. That is a serious publication. A restaurant or a hotel earns pennies.

The numbers back it up: of everything in our record, only 1,848 domains publish a licence or a price at all. If tolls were lucrative, that figure would not look like that.

Two things, and the gap between them. First, what your site declares: your robots.txt, any licence terms, any preference signal in your headers. Second, what your site actually does when three named AI crawlers ask it for your homepage.

The second half is the part nobody runs. It is also where the surprises are.

Really, and in both directions. On 6 September 2026 we measured theguardian.com and nytimes.com naming AI crawlers in robots.txt — declaring a block — and then serving all three of our test identities a normal page. A hotel, casamonica.com, declares that it blocks AI crawlers and let all three straight in.

It runs the other way too. stackoverflow.com declares nothing at all about AI, so on paper it is wide open, and in practice it refused all three of them with a 403.

None of this is scandalous. The file that states the policy and the server that enforces it are usually managed by different people at different times, and nobody checks.

Yes, and there is no card and no account. We would rather you found out you were fine than paid us to tell you so. If it comes back with a problem, that is when there is something to buy.

We have swept 449,070 domains and 347,935 of them answered, which is a lot of websites but nowhere near all of them. If yours is not in there, the live half of the check still runs — we still knock on your door and see what happens — and the report says plainly which half is missing.

Cloudflare starts blocking AI training and agent crawlers by default on ad-supported pages, unless the site owner opts out. Cloudflare sits in front of an enormous number of websites — 215,555 of the domains in our own sweep — so a great many sites will change what they tell AI on a single day.

Nobody is going to email you about it. If it affects you, your site's answer changes and everything looks exactly the same from the front.

It depends entirely on which crawlers you wanted. If you were being harvested by training crawlers and did not want to be, the change does you a favour. If you were turning up in AI answers and quietly getting customers from it, a blunt default block can take that away without you connecting the two.

That is the case for knowing where you stand before the date rather than after.

Then this particular change does not touch you, and the check will say so. Your robots.txt still says whatever it says, and it may still not match what your server does.

Anything that is not visible from outside your website. We make two requests — your homepage and your robots.txt — with a user agent that says honestly who we are. We do not read your server logs, so we cannot tell you which crawlers have actually visited you or how often.

We also cannot see a policy that lives somewhere a machine will not look: inside a terms-of-service page, behind JavaScript, or in a private arrangement visible only to verified crawlers. Those all read to us as "declared nothing", which is where 82.4% of answering domains land. That is a real limitation and we would rather state it than let the number look cleaner than it is.

Because that is when we started. We have nothing from before it and we have not filled the gap in from anywhere else.

It makes for a thin history today and a verifiable one, which is the trade we would make every time.

We identify ourselves honestly, with a contact address, and we never pretend to be another crawler. We make exactly two requests per domain.

We do read the homepage regardless of what robots.txt says, because measuring what a site serves is the entire point of the exercise. If you would rather we did not include your domain, email us and we will take it out.

No. What we record about your site is what your site publishes to anybody who asks, so there is no private information in it to sell.