CrawlState Site Review
What your website
gives away to AI
One page, about one website. What it tells AI crawlers, what those crawlers are actually served when they arrive, and whether the change coming on 15 September 2026 lands on you. Read from the record and measured live. Nothing is estimated.
Try one: theguardian.comnytimes.comreddit.com
The report reads your robots.txt policy out of the CrawlState record — 449,070 domains swept on 6 September 2026, 347,935 of which answered — and then measures what your site does right now when three named AI crawlers ask it for your homepage. A domain that is not in the record still gets the live measurement, and the report says plainly which half is missing.
Every review lives at its own address, so crawlstate.com/review/yourdomain.com can be sent to someone as it is. The awkward questions · How a verdict is decided