ARAIgcRobotsCrawler controls

Anthropic crawler rules

Write ClaudeBot rules for training and user-requested fetches

Keep ClaudeBot, Claude-User, and Claude-SearchBot separate so the file matches how you want Anthropic crawlers to handle your site.

Generate Claude crawler rules

Suggested rule

  • Block ClaudeBot if model-training use is not desired.
  • Allow search and user-triggered tokens only when those fetches are useful for your site.
  • Use server logs to confirm real traffic patterns after publishing.

Official reference

Re-check Claude support's current crawler guidance before publishing production rules: Anthropic crawler guidance.