Free browser tool
Choose how GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, and other crawlers should handle your site. Nothing is uploaded.
Enter your public domain and choose the first rule set.
One path per line. These are added under User-agent: *.
Decide which crawler tokens are allowed or blocked.
Copy this into the root of the exact host you want to control.
Robots.txt validator
Publish check
Generate rules to see the checks.
Robots.txt tells crawlers which paths they should avoid. Some AI and search products publish their own user-agent names, so you can write separate rules for training, search, and user-requested fetches. It is public guidance, not access control.
For most public sites, keep normal search crawlers open, block account or checkout paths, and decide separately whether training crawlers can read public pages. The default preset follows that pattern.
Crawler names and provider rules change. Check the official docs before replacing a production robots.txt file, especially on publisher, SaaS, or ecommerce sites.
Use these provider pages to verify crawler names and policy behavior before changing production robots.txt.
Guides and examples
Use these short guides when you need to review a policy before changing production robots.txt.
Block training access while keeping search and user-requested fetches separate.
Open guideAvoid blocking Google Search when you only want Gemini-related control.
Open guideReview common crawler tokens by training, search, and user-triggered use.
Open guideCompare common policies before generating and publishing your own file.
Open examplesNot always. A full block can stop training bots, but it can also block search or user-requested fetches. Use separate rules when you still want search traffic.
Google describes Google-Extended as a Gemini-related control token, not a Google Search ranking signal. Keep Googlebot open if you rely on organic search.
No. Sensitive data must be protected with authentication, authorization, and server-side access controls. Robots.txt is a crawl preference file.
No. The generator, validator, and publish checks run in your browser.