SEO Log File Analyzer

See how Google, Bing and AI crawlers really crawl your site, from your own server logs: verified Googlebot and Bingbot (fakes separated out), crawl waste on parameters, redirects and errors, the pages Google crawls most, sitemap URLs it never visits, and what GPTBot, ClaudeBot and PerplexityBot fetch.

python3 -m crawllog analyze access.log access.log.1.gz --sitemap sitemap.xml --csv reports
python3 -m crawllog analyze logs/*.gz --json > crawl.json
python3 -m crawllog update-ranges      # refresh Google's and Bing's published crawler IP lists

• Verified crawlers: Googlebot (smartphone, desktop, Image, Video, News), AdsBot, Google-InspectionTool, Storebot, GoogleOther and Bingbot are checked against the IP ranges Google and Bing publish (snapshots included, dated Nov 9, 2026 for Google). Requests that claim to be Googlebot from other IPs are counted as fake and listed with their IPs. Optional --rdns runs Google's reverse + forward DNS check too.
• Crawl budget: verified Googlebot hits by day and by site section, most crawled URLs with last crawl date and status, and the share of requests going to parameter URLs, redirects, 4xx, 5xx, CSS/JS/images and robots.txt.
• Sitemap coverage: sitemap URLs Googlebot didn't crawl in the period, and 200 pages it crawled that aren't in the sitemap.
• AI and other bots: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot, Applebot, SEO tools (AhrefsBot, SemrushBot...) with their top sections.
• CSV reports (--csv DIR): bots.csv, urls.csv, daily.csv, fake.csv, sitemap-not-crawled.csv, crawled-not-in-sitemap.csv for spreadsheets.

Apache and Nginx combined or common log format (extra fields after the user agent are fine), JSON lines with common field names (Nginx JSON, Cloudflare Logpush ClientIP/ClientRequestURI/EdgeResponseStatus...), and gzip files. Lines it can't read are counted and sampled so you can see why.

skills/crawl-log-analysis/SKILL.md tells Claude Code (or any agent that reads skills) how to run the analysis and explain it. Copy the folder into .claude/skills/; the kit is also a Claude Code plugin (.claude-plugin/plugin.json).

Run ./demo.sh for the example (a week of logs for a small shop: 443 lines, fake Googlebots, a 503, a 404 Google keeps crawling, 3 sitemap URLs never crawled) and ./run_tests.sh for the tests. Python 3.9+, no dependencies; everything runs on your computer (only update-ranges and --rdns use the network).

Crawler identification uses the user agent plus, for Google and Bing, published IP ranges; other bots aren't IP-verified, so their counts can include impostors. Bing's published list is dated 2024 (Bing's current file); use --rdns if Bingbot shows up as fake. It reads the logs you give it: if a CDN serves cached pages, those requests are only in the CDN's logs. It doesn't tell you what Google indexed; compare with Search Console.

Pairs with the SEO Audit MCP Server. License: LICENSE.txt.