AI robots.txt generator

Draft robots.txt rules for AI crawlers

Enter a host, choose the crawlers to block or allow, and copy a robots.txt file you can review before publishing.

Open the generator Validate robots.txt

What it generates

A general User-agent: * section for private paths such as account, checkout, API, and admin URLs.
Specific block rules for AI-related crawlers when you choose a restrictive policy.
A sitemap declaration using the host you enter.

How to use it

Enter the exact host you want to control, such as https://example.com.
Add private or transactional paths that should not be crawled broadly.
Choose a preset, review each crawler, then copy or download the generated file.
Run the validator before publishing the file at /robots.txt.

When to use it

Use it when launching or updating a content site, SaaS marketing site, ecommerce site, or technical blog before you do deeper log analysis.

Common mistakes to avoid

Blocking Googlebot by accident on a site that relies on Google Search.
Using Disallow: / under User-agent: * without understanding the impact.
Forgetting that robots.txt is not a security boundary for private data.