ARAIgcRobotsCrawler controls

Crawler guide

Google-Extended is not the same as Googlebot

Use Google-Extended for Gemini-related control, and keep Googlebot open when normal Google Search traffic matters.

Generate Google rules Run publish check

What each token means

  • Googlebot is the crawler used for normal Google Search crawling.
  • Google-Extended is a robots.txt control token for Gemini-related use cases.
  • Blocking one token should not be treated as blocking all Google access.

Common mistake

A generic AI crawler blocklist may include Googlebot. That can reduce search visibility on sites that rely on organic traffic. Review Google-specific rules separately.

Safer workflow

  • Block Google-Extended only if that matches your AI data-use policy.
  • Keep Googlebot open for public pages unless deindexing is intentional.
  • Validate the final file and submit the live URL to Search Console after deployment.