Search open, training blocked
This is the safer default for many public sites: keep normal search crawlers and useful search fetchers
open, block training-oriented tokens, and protect private paths under User-agent: *.
Examples
Robots.txt examples for AI crawler policies
Use these examples to compare common policies before generating and validating your own file.
Private paths for all crawlers
Common paths include /admin/, /account/, /checkout/,
/api/, and /private/. Robots.txt is only guidance, so sensitive data still needs
server-side protection.
Strict access policy
A stricter policy may block most AI crawler tokens. Use it deliberately because it can also reduce visibility in AI search products and referral surfaces.