Robots.txt Tester.
Validate robots.txt. Simulate every crawler.
Paste your robots.txt — or just a domain — and we'll parse it against the full spec, simulate how Googlebot, Bingbot, GPTBot, ClaudeBot, and 30 other crawlers will read it, and flag every conflict, ambiguity, or accidental disallow.
Bookmark-worthy moments.
Four moments when a single line in robots.txt is the difference between visibility and invisibility.
Test a new disallow rule before you ship
About to block /admin/*? Paste the proposed rule, simulate against Googlebot, confirm nothing important matches. Catch the wildcard mistake before it deindexes a section.
Debug missing-from-index pages
GSC says a URL is 'blocked by robots.txt' but you can't see why. Run the URL through the simulator, see which exact line matches, fix it.
Control AI crawler access
Want to allow Googlebot but block GPTBot, ClaudeBot, and CCBot? See if your current rules actually do that — or if you're accidentally letting them all through.
Audit a deployment that just went live
Engineering pushed a robots.txt change. Verify staging-style disallows weren't shipped to production. Confirm sitemap directives still point at the right XML files.
How robots.txt actually parses.
The spec quirks that catch out everyone — wildcards, longest-match-wins, group-merging, and the AI-crawler explosion of 2024.
robots.txt is one of those files that looks trivial until something goes wrong. The spec is small but full of edge cases: rules are case-sensitive, wildcards (* and $) work in ways that aren't intuitive, multiple groups for the same user-agent get merged, and the 'longest matching rule wins' tiebreaker overrides the order rules appear in the file. Most teams write a rule, test it on one URL, and assume it works for everything.
Then the AI crawlers happened. Since 2023, the robots.txt landscape has gotten meaningfully more complex. There are now 30+ named AI crawlers — GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Bytespider, PerplexityBot, and counting — each of which respects robots.txt independently from search-engine crawlers. Blocking 'Googlebot' does nothing to GPTBot. Allowing CCBot means your content is in Common Crawl, which means it's in everyone's training data.
This tool parses your robots.txt against the official spec, then runs your URLs through the same matching logic each crawler uses. You see exactly which rule (line number, rule text) matched for each crawler, and what the resulting decision is — allow or disallow. No more guessing.
Directives + crawlers · quick reference
Which crawler the following rules apply to. * = all crawlers (lowest priority).
User-agent: GPTBotExplicitly permit a URL pattern. Wins ties via longest-match rule.
Allow: /public/Block a URL pattern. Empty value (Disallow:) means 'allow all'.
Disallow: /admin/Where your XML sitemap lives. Global (not user-agent specific). Multiple lines allowed.
Sitemap: https://x.com/sitemap.xmlRequested seconds between crawls. Ignored by Googlebot; respected by Bingbot.
Crawl-delay: 10Match any sequence of characters within a path segment.
Disallow: /*.pdf$Match end of URL. Useful for blocking file extensions specifically.
Disallow: /search?*OpenAI's training crawler. Block to opt out of GPT training corpus.
User-agent: GPTBotAsked. Answered.
Tool questions, topic questions — answered without the hedging. Anything missing? Email hi@seonova.io and we'll fold it in.
For crawling, yes — Googlebot strictly respects Disallow directives. For indexing, almost — a URL blocked by robots.txt can still be indexed if Google discovers it via external links (the snippet will read 'No information available'). To prevent indexing, use a noindex meta tag, not robots.txt.
More to bookmark.
Tools that share the same engine — and tend to land in the same tab group.
Catch robots.txt drift before it deindexes you.
SEONova Pro watches your robots.txt for changes every hour, diffs every edit, and pages the SEO lead when a rule that wasn't there yesterday blocks Googlebot today. The free tool tests once. The platform watches forever.