Enter a domain to see which AI and search crawlers your robots.txt allows, and whether your server actually lets them in. The tool checks 19 crawlers, including GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Bingbot, then requests your page the way a crawler would.
Reads your robots.txt and makes one request to the page as GPTBot, to catch sites that allow AI crawlers on paper but block them in practice.
Why robots.txt is only half the answer
Most checkers stop at reading robots.txt. That misses the two most common reasons an AI assistant cannot read a site that appears to allow it.
The first is a firewall. robots.txt is a polite request, not a lock. A CDN bot rule, a security plugin or a hosting firewall can refuse crawlers at the network level while robots.txt says everyone is welcome. Cloudflare, for one, has blocked AI crawlers by default on new domains since July 2025, and site owners often do not know the setting exists.
The second is JavaScript. If your page is an empty shell that fills in after scripts run, a crawler that does not execute JavaScript sees nothing. Most AI crawlers do not. The page loads, returns 200, and contains no words.
So this tool does both: it parses your robots.txt properly, then makes one request to the page identifying itself as GPTBot and reports the status code and how many words are in the HTML before any script runs.
How to read the result
| You see | It means | What to do |
|---|---|---|
Blocked, with a rule from User-agent: * |
A general rule is catching this crawler | Add a specific group for the crawler if you want it allowed |
| Blocked, with its own rule | Someone blocked this crawler deliberately | Find out who and why before changing it |
| Allowed, but HTTP 403 on the page | Something other than robots.txt is refusing crawlers | Check bot settings in your CDN and security plugin |
| HTTP 200 but very few words | Content needs JavaScript to appear | Render the main content on the server |
| robots.txt returns 5xx | Google treats this as “crawl nothing” | Fix urgently |
Training crawlers and search crawlers are different
The table labels each crawler by purpose because the decision is not all or nothing. GPTBot and ClaudeBot collect text for model training. OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes that assistants search when answering a question. ChatGPT-User and Claude-User fetch a single page because a person asked about it.
A business that wants to be recommended by assistants should allow the search and user crawlers. Whether to allow the training crawlers is a separate choice, and reasonable people land on both sides. Blocking everything with “AI” in the name, which is what many security plugins do in one click, removes you from the answers as well as the training data.
What this tool cannot tell you
Our request comes from this website’s server, not from OpenAI’s network. Some firewalls verify a crawler’s address and block anything that claims to be GPTBot from elsewhere, while still letting the real one through. So a 403 here is a reason to look at your firewall settings, not proof that GPTBot is blocked. The reliable confirmation is your own server log: search it for the crawler names and see what status codes they received.
The tool also tests one path. A rule can allow your home page and block your blog. Enter a full page URL to test that page.
Frequently asked questions
I have no robots.txt. Is that a problem?
No. A missing robots.txt means everything may be crawled. It is worth adding one so you can list your sitemap, but its absence blocks nothing.
Why is Google-Extended in the list if it is not a crawler?
Google-Extended is a control token. Googlebot does the fetching, and the Google-Extended rule decides whether what it fetched may be used for Gemini. Blocking it does not affect Google Search.
Will allowing these crawlers slow my site down?
Rarely. If one crawler is too aggressive, block or rate limit that one rather than all of them.
Related
- AI crawlers and robots.txt: which bots to allow and how to write the rules
- Machine readability and clean HTML: fixing pages that need JavaScript
- Sitemap Freshness Auditor: the next thing a crawler reads
- All free SEO tools