robots.txt Tester (Free)

Test any URL against a robots.txt the way Google documents it (RFC 9309: longest matching rule wins, ties go to allow, * and $ wildcards, percent-encoding, one group per crawler), lint the file, see which AI crawlers can fetch your site, and diff a new robots.txt against the live one before you deploy it.

python3 robots_check.py test robots.txt /blog/post /admin --ua Googlebot
python3 robots_check.py test robots.txt --urls urls.txt --ua Googlebot,Bingbot,GPTBot
python3 robots_check.py lint robots.txt
python3 robots_check.py ai robots.txt
python3 robots_check.py diff live.txt new.txt --urls important-urls.txt     # exit 1 if any URL becomes blocked
python3 robots_check.py fetch https://example.com/robots.txt -o live.txt    # optional download

Everything except fetch runs offline. --json on every command. Python 3.9+, standard library only. ./demo.sh shows each command on the examples; ./run_tests.sh runs the tests, which include every example table from Google's robots.txt documentation.

ALLOW / BLOCK, the exact rule and line number that decided it, and which group the crawler used. Crawler groups follow Google's rules: the most specific user-agent group wins, groups for the same crawler are merged, and * is not combined with a crawler's own group. Googlebot-Image, -Video, -News and Google-InspectionTool fall back to a Googlebot group, as listed in Google's crawler docs.

• Disallow/Allow before any User-agent (ignored), typos (User agent, Dissallow, Sitemaps), lines without a colon
• full URLs in Disallow, paths not starting with /, $ in the middle of a path, spaces in paths
• Sitemap that isn't an absolute URL; no sitemap at all
• a crawler with its own group silently dropping your * rules (the most common robots.txt mistake)
• Disallow: / for * (staging file shipped to production?)
• unsupported fields: Noindex, Crawl-delay (ignored by Google), Host, Clean-param
• HTML served as robots.txt, files over Google's 500 KiB limit, byte order marks, duplicate groups, allow/disallow on the same path

Shows ALLOW/BLOCK for / (or --path) for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider, MistralAI-User and DuckAssistBot, and whether each one is governed by its own group or by *. Google-Extended and Applebot-Extended are control tokens, not crawlers. User-triggered fetchers (ChatGPT-User, Perplexity-User and others) may not follow robots.txt at all, so blocking them here is a request, not a guarantee.

• It reads a robots.txt file; it doesn't know how your server answers (a 4xx for robots.txt means "no restrictions" to Google; 5xx pauses crawling).
• Blocking crawling doesn't remove a URL from search results; use noindex.
• Other search engines differ in details (Bing and Yandex read Crawl-delay).