ARAIgcRobotsCrawler controls

Robots.txt validator

Check robots.txt before publishing

Paste a file or use the generated output to catch missing user-agent groups, duplicate crawler entries, site-wide blocks, and missing sitemaps.

Open the validator Generate rules first

What this validator can catch

  • Rules placed before a User-agent group.
  • Repeated crawler groups that make audits harder.
  • User-agent: * combined with Disallow: /.
  • Missing sitemap declarations and weak protected path coverage.

When to run it

  • Before publishing a new /robots.txt file.
  • After adding AI crawler rules for GPTBot, ClaudeBot, Google-Extended, or PerplexityBot.
  • Before submitting a site to Search Console or launching a new content section.

What it does not replace

This checker does not fetch live URLs, verify server status codes, or replace Google Search Console and provider-specific crawler tools.

After validation

After the validator is clean, check whether Googlebot is still open and whether your sitemap is listed.