robots.txt Generator
Generate a robots.txt with disallow and allow rules, sitemap lines, a block-everything option for staging and switches to opt out of AI training crawlers.
Free · No sign-up · Your inputs stay in your browser
Input
Generate a robots.txt
Block AI crawlers
Output
robots.txt
146 characters
robots.txt (RFC 9309) controls crawling, not indexing: a disallowed URL can still appear in results if other pages link to it, and Google cannot see a noindex tag on a page it may not crawl. Use noindex to keep a page out of search. The longest matching rule wins, so a more specific Allow beats a shorter Disallow. Blocking Google-Extended keeps content out of Gemini training without affecting Search; a staging site needs authentication, not just this file.
Runs in this browser. Nothing you paste is sent to a server.
About this tool
Method, assumptions, practical next steps, and limits.
Crawling, not indexing
robots.txt, standardised in RFC 9309, tells crawlers what not to fetch. It does not remove pages from search: a disallowed URL can still be indexed from links pointing to it, and Google cannot see a noindex tag on a page it is not allowed to crawl. To keep a page out of results, allow crawling and use noindex.
AI training crawlers
Google-Extended controls whether Google uses content to train Gemini without affecting Search, and Applebot-Extended does the same for Apple. Other AI companies publish their crawler names; robots.txt only stops crawlers that honour it.