robots.txt tester
Check which URLs a robots.txt allows or blocks for a crawler, with the exact rule that decided each one, following RFC 9309.
A clear answer. Here’s how.
Parses the file as RFC 9309 section 2 describes: user-agent lines open a group, consecutive user-agent lines share it, and allow and disallow rules belong to the group above them; other records such as Sitemap do not end a group (section 2.2.4), and rules before any user-agent line are ignored. The crawler follows every group whose product token equals its own, ignoring case, merged into one (section 2.2.1); only if there is none does it follow the * groups. For each URL the path and query are compared, case-sensitively, with every rule after percent-encoding normalisation (non-ASCII as UTF-8 escapes, escaped unreserved characters decoded, hex in upper case): * matches any run of characters and a final $ anchors the end (section 2.2.3). The longest matching rule wins, allow wins a tie, and no match means allowed; /robots.txt itself is always allowed (section 2.2.2). The deciding rule and its line are shown. Warnings cover a byte order mark, rules outside a group, unknown or misspelt directives, unreadable user-agent values and files over 500 KiB. Crawl-delay and Sitemap are listed but not applied. Reserved characters such as : or / inside a query are compared as written, so a rule must use the same escaping as the URL; the host of a full URL is not compared, and Google’s crawler-specific fallbacks (such as Googlebot-Image using the Googlebot group) are not modelled.
How this is checked
5 reference checks passed These 5 cases were run when this page was built, on 20 October 2026, and all reproduce within the stated tolerance. The same cases are part of the automated tests that run on every change to Toolmoz. Engine version 1.
See the checks and where each expected answer comes from
Case 1
- Inputs
- robots.txt user-agent: * allow: /p disallow: /Crawler product token GooglebotURLs or paths to test, one per line https://example.com/page
- Expected, and what Toolmoz gave
- Verdict (first URL) expected Allowed, Toolmoz AllowedDeciding rule (first URL) expected allow: /p, Toolmoz allow: /p
- Where the expected answer comes from
- Google, Order of precedence for rules: for https://example.com/page, allow: /p applies because it is more specific than disallow: /
- Allowed difference
- exact
Case 2
- Inputs
- robots.txt user-agent: * allow: /folder disallow: /folderCrawler product token GooglebotURLs or paths to test, one per line https://example.com/folder/page
- Expected, and what Toolmoz gave
- Verdict (first URL) expected Allowed, Toolmoz AllowedDeciding rule (first URL) expected allow: /folder, Toolmoz allow: /folder
- Where the expected answer comes from
- Google, Order of precedence for rules: conflicting rules of equal length, so the least restrictive (allow: /folder) applies; RFC 9309 section 2.2.2: allow SHOULD be used when equivalent
- Allowed difference
- exact
Case 3
- Inputs
- robots.txt user-agent: * allow: /page disallow: /*.htmCrawler product token GooglebotURLs or paths to test, one per line https://example.com/page.htm
- Expected, and what Toolmoz gave
- Verdict (first URL) expected Blocked, Toolmoz BlockedDeciding rule (first URL) expected disallow: /*.htm, Toolmoz disallow: /*.htm
- Where the expected answer comes from
- Google, Order of precedence for rules: disallow: /*.htm applies to /page.htm because the rule path is longer
- Allowed difference
- exact
Case 4
- Inputs
- robots.txt user-agent: * allow: /$ disallow: /Crawler product token GooglebotURLs or paths to test, one per line https://example.com/page.htm
- Expected, and what Toolmoz gave
- Verdict (first URL) expected Blocked, Toolmoz BlockedDeciding rule (first URL) expected disallow: /, Toolmoz disallow: /
- Where the expected answer comes from
- Google, Order of precedence for rules: allow: /$ only applies to the root URL, so disallow: / applies to /page.htm
- Allowed difference
- exact
Case 5
- Inputs
- robots.txt user-agent: ExampleBot disallow: /foo disallow: /bar user-agent: ExampleBot disallow: /bazCrawler product token ExampleBotURLs or paths to test, one per line /baz/index.html
- Expected, and what Toolmoz gave
- Verdict (first URL) expected Blocked, Toolmoz BlockedDeciding rule (first URL) expected disallow: /baz, Toolmoz disallow: /bazRules in the group expected 3, Toolmoz 3
- Where the expected answer comes from
- RFC 9309 section 2.2.1 figure 2: two ExampleBot groups merge into one group with disallow /foo, /bar and /baz
- Allowed difference
- exact
What this shows: the engine reproduces independently worked values for these cases. It does not prove every possible input is right. To test an answer you were given, see how to check a number; to see what is and is not checked across Toolmoz, see the accuracy ledger.
See a worked example
Worked example
A fixed demonstration, separate from the values you enter above.
- robots.txt
User-agent: * Disallow: /private/ Allow: /private/public-note.html User-agent: Googlebot Disallow: /*.php$ Allow: /page Disallow: /*.ph Crawl-delay: 5 Sitemap: https://example.com/sitemap.xml
- Crawler product token
- Googlebot
- URLs or paths to test, one per line
/index.php /index.php?lang=en https://example.com/page.php5 /private/report.pdf /private/public-note.html
Example result
| URL | Path compared | Verdict | Deciding rule | Why |
|---|---|---|---|---|
| /index.php | /index.php | Blocked | Line 6: disallow: /*.php$ | Longest matching rule (7 characters) |
| /index.php?lang=en | /index.php?lang=en | Blocked | Line 8: disallow: /*.ph | Longest matching rule (5 characters) |
| https://example.com/page.php5 | /page.php5 | Allowed | Line 7: allow: /page | Longest matching rule (5 characters); an allow and a disallow of equal length match, and allow wins the tie |
| /private/report.pdf | /private/report.pdf | Allowed | – | No rule in the group matches, so it is allowed (rules in * groups do not apply to a crawler that has its own group) |
| /private/public-note.html | /private/public-note.html | Allowed | – | No rule in the group matches, so it is allowed (rules in * groups do not apply to a crawler that has its own group) |
Matching follows RFC 9309: the group for the exact product token (case-insensitive), or the * groups, merged; the longest matching rule wins and allow wins a tie; * matches any characters and a final $ anchors the end. Paths are case-sensitive and compared after percent-encoding normalisation. The host of a full URL is not compared, because a robots.txt applies only to the host it came from. Sitemaps: https://example.com/sitemap.xml. Crawl-delay is shown but not applied; Google ignores it.