robots.txt Tester
Test URLs against a robots.txt for any crawler, including Googlebot and AI crawlers, and see the exact rule that allows or blocks each one.
Free · No sign-up · Your inputs stay in your browser
Input
Test URLs against robots.txt
Output
Group: *
| Path | Result | Deciding rule |
|---|---|---|
| /library/laravel-queues | Allowed | no matching rule |
| /search?q=redis | Blocked | Disallow: /search |
| /search/help | Allowed | Allow: /search/help |
| /files/guide.pdf | Blocked | Disallow: /*.pdf$ |
| /files/guide.pdf?v=2 | Allowed | no matching rule |
| Sitemap 1 | https://example.com/sitemap.xml |
|---|
Matching follows RFC 9309, as Google implements it: a crawler obeys only the most specific group naming it, the longest matching rule wins, * matches anything and $ anchors the end. robots.txt controls crawling, not indexing: a blocked URL can still appear in results if other pages link to it. Use noindex for that, on a page that is not blocked.
Runs in this browser. Nothing you paste is sent to a server.
About this tool
Method, assumptions, practical next steps, and limits.
How matching works
RFC 9309, published in 2022, standardised what Google had long done. A crawler obeys only the most specific group that names it, so a User-agent: GPTBot group replaces the * rules for GPTBot rather than adding to them. Within the group the longest matching path wins, which is how Allow: /search/help beats Disallow: /search.
Blocking is not hiding
robots.txt controls crawling, not indexing: a blocked URL can still appear in results, without a description, when other pages link to it. To keep a page out of an index, let it be crawled and send noindex. And never list private paths in robots.txt; it is public, and it tells everyone where they are.