Implementation guide

How to block GPTBot without breaking useful search access

GPTBot is only one OpenAI crawler token. Treat it separately from search fetchers and user-requested retrieval before publishing robots.txt.

Generate GPTBot rules Open examples

Recommended baseline

Block GPTBot if your policy does not allow public pages to be used for model training.
Keep OAI-SearchBot open only when ChatGPT search visibility is useful for your site.
Do not block Googlebot unless you intentionally want to reduce Google Search crawling.

Example rule

User-agent: GPTBot Disallow: /

This blocks GPTBot broadly. Add separate general rules for account, checkout, API, and private paths under User-agent: *.

Before publishing

Run the validator to catch broad User-agent: * mistakes.
Check OpenAI's current crawler documentation because token names and behavior can change.
Monitor server logs after deployment to confirm the rules match real crawler traffic.