Log File Analysis at Scale With Python: Finding Crawl Waste

Log file analysis shows exactly which URLs Googlebot requests on your site, how often, and with what result, which no other tool can tell you. At scale, the useful questions are: which sections take most crawls, how many requests go to parameter URLs, which status codes Googlebot receives, and which important pages it never visits. A short Python script with pandas can answer all four from a standard access log, and comparing the result with your sitemap shows where crawling is wasted.

What logs reveal that other tools do not

Step 1: get the logs

Download access logs from your hosting control panel, server or CDN for at least two to four weeks. The script expects the common “combined” log format used by Apache and Nginx. Large sites may need to export logs from a CDN or log service.

Step 2: the script

#!/usr/bin/env python3
"""Summarise Googlebot crawling from an access log: where crawls go and where they are wasted.
Usage: python3 crawl_waste.py access.log [sitemap_urls.txt]"""
import re, sys
from urllib.parse import urlsplit, parse_qs
import pandas as pd

LINE = re.compile(r'^(\S+) \S+ \S+ \[([^\]]+)\] "(?:GET|HEAD) (\S+) [^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')

rows = []
for line in open(sys.argv[1], errors="replace"):
    m = LINE.match(line)
    if m and "googlebot" in m.group(5).lower():
        url = m.group(3); parts = urlsplit(url)
        rows.append({"path": parts.path, "params": ",".join(sorted(parse_qs(parts.query))),
                     "section": "/" + (parts.path.strip("/").split("/")[0] if parts.path.strip("/") else ""),
                     "status": m.group(4), "has_query": bool(parts.query)})
df = pd.DataFrame(rows)
print(f"Googlebot requests: {len(df)}\n")
print("By section:\n", df["section"].value_counts().head(15).to_string(), "\n")
print("By status:\n", df["status"].value_counts().to_string(), "\n")
q = df[df.has_query]
print(f"Requests with query strings: {len(q)} ({len(q)/max(len(df),1):.0%})")
if len(q): print("Most crawled parameters:\n", q["params"].value_counts().head(10).to_string(), "\n")
print("Most crawled URLs:\n", df["path"].value_counts().head(10).to_string(), "\n")
if len(sys.argv) > 2:
    sitemap = {urlsplit(u.strip()).path for u in open(sys.argv[2]) if u.strip()}
    crawled = set(df["path"])
    print(f"Sitemap URLs never crawled: {len(sitemap - crawled)}")
    print(f"Crawled paths not in sitemap: {len(crawled - sitemap)}")

Install pandas with pip install pandas. The second argument is a text file with one sitemap URL per line.

Step 3: read the results

We ran the script on a synthetic log shaped like a typical online shop, with product pages, sort and colour filters, internal search and some removed pages. The key lines were:

Requests with query strings: 145 (72%)
Most crawled parameters:
color,sort    120
q              25
Sitemap URLs never crawled: 21

In that pattern, nearly three quarters of Googlebot’s requests went to filter and search URLs while a third of the sitemap’s product pages were never requested. On a real site, that is the signal to block low-value parameters, stop linking to internal search results, and strengthen internal links to the missed pages. See crawl budget explained.

Verify the crawler

The script trusts the user-agent string, which anyone can fake. For decisions that matter, confirm requests with reverse DNS or published IP ranges, using the method and script in verifying Googlebot and GPTBot in server logs.

Extending it

• Change the user-agent filter to analyse Bingbot, OAI-SearchBot, PerplexityBot or Claude-SearchBot.
• Group by date to see crawl trends before and after a release.
• For very large logs, read in chunks with pandas or load into a database such as DuckDB or BigQuery.

Frequently asked questions

How much log data is enough?

Two to four weeks shows a reliable pattern for most sites. Larger sites may see enough in a few days.

Our host does not give us logs. What then?

Check whether your CDN offers logs, or ask the host. Without logs, Search Console’s Crawl stats report is the best alternative.

Are logs personal data?

Logs contain IP addresses, which can be personal data. Keep them secure and only as long as needed, and remove visitor IPs if you only analyse crawlers.

Sources

1. Google Search Central, “Large site owner’s guide to managing your crawl budget”
2. Google Search Central, “Verifying Googlebot and other Google crawlers”
3. pandas documentation