ARAIgcRobotsCrawler controls

OpenAI crawler rules

Write GPTBot rules without blocking OpenAI search fetches

GPTBot, OAI-SearchBot, and ChatGPT-User are separate tokens. You can block training access while leaving search or user-requested fetches open.

Generate OpenAI crawler rules

Suggested rule

  • Allow OAI-SearchBot if ChatGPT search traffic is useful.
  • Block GPTBot when you do not want public pages used for model-training datasets.
  • Treat ChatGPT-User as user-triggered retrieval, not automatic crawling.

Official reference

Re-check OpenAI's current crawler documentation before publishing production rules: OpenAI crawlers.