Robots.txt Generator
Create a valid robots.txt, block AI crawlers if you want, and test URLs against it.
How to use Robots.txt Generator
- Pick a starting point, then add the paths you want to block or allow.
- Add your sitemap URL.
- Test a few URLs, then download robots.txt and upload it to your site’s root (example.com/robots.txt).
How it works
robots.txt follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022. Crawlers read the group that matches their name (or the * group), and the longest matching rule wins; when an Allow and a Disallow match equally, Allow wins. The tester applies exactly these rules, including * wildcards and $ end anchors.
robots.txt controls crawling, not indexing: a blocked page can still appear in results if other sites link to it. To keep a page out of search results, let it be crawled and add a noindex meta tag instead. The AI preset lists crawlers that vendors have documented for training or AI search; names change, so check vendors’ documentation, and remember that robots.txt is a request that well-behaved bots honour, not an enforcement mechanism.
Frequently asked questions
Does Disallow remove a page from Google?
No. It stops crawling, not indexing. Use a noindex meta tag (and keep the page crawlable) to remove it from results.
How do I block AI crawlers but stay in Google Search?
Choose "Allow search, block AI training crawlers". It blocks agents such as GPTBot, ClaudeBot, CCBot and Google-Extended, while Googlebot can still crawl for search.
Where does robots.txt go?
At the root of each host: https://example.com/robots.txt. Subdomains need their own file.
Is Crawl-delay supported?
Google ignores it. Some other search engines honour it. It is omitted here to avoid false expectations.
Limitations
- Rules are case-sensitive for paths.
- The AI crawler list reflects vendor documentation as of November 2026 and will need updates.
Last reviewed November 9, 2026. How we build and test tools.