ARAIgcRobotsCrawler controls

AI crawler rules

Choose which crawler tokens to allow or block

Set crawler tokens one by one instead of pasting a generic blocklist that may block useful search traffic.

Build rules

Why selective rules matter

Training crawlers, search crawlers, and user-triggered fetches do different jobs. Separate rules make the file easier to review before publishing.

Simple workflow

  • Keep normal search crawlers open unless you intentionally want deindexing.
  • Block private, transactional, and account paths for all crawlers.
  • Decide separately whether AI training crawlers can access public content.