Checks included
- Sitemap declared for the current host.
- Private paths blocked for general crawlers.
- Googlebot is not accidentally blocked.
- Search crawler rules are set deliberately.
Publish check
Review robots.txt before it goes live
Check sitemap coverage, private path rules, Googlebot access, training-crawler blocks, and search crawler rules in one local checklist.
What the score checks
The score is a checklist, not a ranking promise. It flags rules that block normal search, omit a sitemap, or block every search-related crawler token.
Who should run it
- Publishers deciding which AI crawlers should read public pages.
- SaaS and ecommerce teams protecting account, checkout, and API paths.
- Developers reviewing robots.txt changes before deployment.
Simple workflow
Generate rules, validate obvious mistakes, then run this check before uploading the file.