How to Verify Real Googlebot and GPTBot Visits in Your Server Logs (Free Script)

To verify that a visit claiming to be Googlebot or GPTBot is genuine, check its IP address, not its user agent, because anyone can copy a user agent. For Googlebot and Bingbot, run a reverse DNS lookup on the IP and confirm the hostname ends in googlebot.com, google.com or search.msn.com, then run a forward lookup to confirm it points back to the same IP. For OpenAI’s crawlers, compare the IP with the ranges OpenAI publishes. The free script below does both for a whole access log.

Why verification matters

Scrapers routinely pretend to be well-known crawlers so that firewalls let them through. If you count every line that says “GPTBot” as OpenAI, your numbers will be wrong, and if you allow any request with that user agent through your firewall, you allow the fakes too. Verification tells you three things: whether the real crawlers visit, what status codes they receive, and how much of your “bot traffic” is impersonation.

Step 1: get your access log

• cPanel hosting: open “Raw Access”, download the log for your domain and unzip it.
• Nginx or Apache on your own server: usually /var/log/nginx/access.log or /var/log/apache2/access.log.
• Behind Cloudflare: your server sees Cloudflare’s IPs unless the real visitor IP is restored from the CF-Connecting-IP header. Check that your log shows real IPs before verifying anything.

Step 2: verify Googlebot and Bingbot with DNS

Google’s documentation describes this method. On Linux or macOS:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
# Hostname ends in googlebot.com and points back to the same IP: genuine

For Bingbot, the hostname should end in search.msn.com. Google also publishes its crawler IP ranges as JSON files, which is faster for large logs.

Step 3: verify OpenAI’s crawlers with published IP ranges

OpenAI’s crawler documentation links to JSON files listing the IP ranges used by GPTBot, OAI-SearchBot and ChatGPT-User. Download the current files from that page, since the ranges change. A request that claims to be OAI-SearchBot but comes from outside those ranges is not OpenAI. For other AI companies, check each company’s crawler documentation for published ranges.

Step 4: run the whole log through the script

This script counts requests from the main search and AI crawlers, lists their status codes, and verifies Googlebot and Bingbot by DNS and OpenAI’s crawlers by IP range. It needs Python 3 and nothing else. Save it as verify_bots.py:

#!/usr/bin/env python3
"""Count search and AI crawler visits in an access log and check which are genuine.
Usage: python3 verify_bots.py access.log [openai-ranges.json ...]"""
import ipaddress, json, re, socket, sys
from collections import Counter, defaultdict

BOTS = ["Googlebot", "bingbot", "OAI-SearchBot", "ChatGPT-User", "GPTBot",
        "ClaudeBot", "Claude-SearchBot", "Claude-User", "PerplexityBot", "Perplexity-User", "CCBot"]
DNS_SUFFIX = {"Googlebot": (".googlebot.com", ".google.com"), "bingbot": (".search.msn.com",)}
LINE = re.compile(r'^(\S+) \S+ \S+ \[[^\]]+\] "[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')

def dns_ok(ip, suffixes):
    try:
        host = socket.gethostbyaddr(ip)[0]
        return host.endswith(suffixes) and ip in socket.gethostbyname_ex(host)[2]
    except OSError:
        return False

def load_ranges(files):
    nets = []
    for f in files:
        for p in json.load(open(f)).get("prefixes", []):
            cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
            if cidr:
                nets.append(ipaddress.ip_network(cidr))
    return nets

def main():
    log, range_files = sys.argv[1], sys.argv[2:]
    nets = load_ranges(range_files)
    hits, statuses, ips = Counter(), defaultdict(Counter), defaultdict(set)
    for line in open(log, errors="replace"):
        m = LINE.match(line)
        if not m:
            continue
        ip, status, ua = m.groups()
        for bot in BOTS:
            if bot.lower() in ua.lower():
                hits[bot] += 1; statuses[bot][status] += 1; ips[bot].add(ip)
                break
    cache = {}
    print(f"{'Crawler':<18}{'Requests':>9}{'IPs':>6}{'Verified':>10}  Status codes")
    for bot in BOTS:
        if not hits[bot]:
            continue
        verified = "-"
        if bot in DNS_SUFFIX:
            ok = sum(cache.setdefault(ip, dns_ok(ip, DNS_SUFFIX[bot])) for ip in ips[bot])
            verified = f"{ok}/{len(ips[bot])}"
        elif bot.startswith(("OAI", "ChatGPT", "GPTBot")) and nets:
            ok = sum(any(ipaddress.ip_address(ip) in n for n in nets) for ip in ips[bot])
            verified = f"{ok}/{len(ips[bot])}"
        codes = ", ".join(f"{c}: {n}" for c, n in statuses[bot].most_common())
        print(f"{bot:<18}{hits[bot]:>9}{len(ips[bot]):>6}{verified:>10}  {codes}")
    print("\nVerified = IPs that passed the check / all IPs using that user agent.")

if __name__ == "__main__":
    main()

Run it with your log and the OpenAI range files:

$ python3 verify_bots.py access.log searchbot.json gptbot.json chatgpt-user.json

We tested it on a small sample log built for the purpose, with fake Googlebot requests from documentation IP addresses and one IP inside a supplied range. The output looked like this:

Crawler            Requests   IPs  Verified  Status codes
Googlebot                 8     2       0/2  200: 8
OAI-SearchBot             4     1       1/1  200: 4
GPTBot                    2     1       0/1  403: 2
PerplexityBot             6     1         -  403: 6

Verified = IPs that passed the check / all IPs using that user agent.

Both “Googlebot” IPs failed verification, as they should, and the script flagged the 403 responses given to GPTBot and PerplexityBot.

Reading the results

Frequently asked questions

My log shows only Cloudflare IPs. What now?

Configure your server to log the original visitor IP from Cloudflare’s CF-Connecting-IP header (Cloudflare documents this for Apache and Nginx), or use Cloudflare’s own security events for bot analysis.

How often should I check?

Monthly, and after any change to hosting, CDN or security settings.

Can I use this on Windows?

Yes, with Python installed. Replace python3 with py if needed.

Sources

1. Google Search Central, “Verifying Googlebot and other Google crawlers”
2. Bing Webmaster Guidelines, “Verifying Bingbot”
3. OpenAI, “Overview of OpenAI crawlers” (IP range files)
4. Cloudflare documentation, “Restoring original visitor IPs”