Most AI companies run several crawlers with different jobs: one collects training data, one builds a search index, and one fetches a page when a user asks about it. robots.txt lets you allow or block each one by name. Blocking the search crawler removes you from that company’s AI answers; blocking only the training crawler usually does not. This page lists the main crawlers and explains how to set rules for them.
The main AI crawlers
| Company | User agent | Purpose |
|---|---|---|
| OpenAI | GPTBot | Collects content that may be used to train models |
| OAI-SearchBot | Builds the index for search results in ChatGPT | |
| ChatGPT-User | Fetches a page when a user’s request needs it | |
| Anthropic | ClaudeBot | Collects content that may be used to train models |
| Claude-SearchBot | Crawls to improve search results in Claude | |
| Claude-User | Fetches a page when a user asks Claude about it | |
| Perplexity | PerplexityBot | Builds Perplexity’s search index |
| Perplexity-User | Fetches a page when a user’s question needs it | |
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode | |
| Google-Extended | A robots.txt token (not a separate crawler) that controls use of content for Gemini training and grounding | |
| Apple | Applebot-Extended | A robots.txt token that controls use of content crawled by Applebot for training Apple’s models |
| Meta | meta-externalagent | Collects content for training and AI products |
| Common Crawl | CCBot | Builds an open web archive that many AI companies train on |
Companies add and rename crawlers from time to time, so check each company’s crawler documentation before making changes.
Three decisions to make
1. Do you want to appear in AI answers?
For most businesses the answer is yes, which means allowing the search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. Google does not offer a separate control for AI Overviews. The only way to keep content out of them is the nosnippet rule, which also removes your normal search snippets.
2. Do you allow model training?
This is a business and rights decision, separate from visibility. You can block GPTBot, ClaudeBot, CCBot and similar crawlers while still allowing the search crawlers above. Content that is already in a training set is not removed by blocking now.
3. What about Google-Extended?
Google-Extended covers Gemini’s use of your content for grounding answers as well as for training. Blocking it may reduce how often Gemini cites you. It does not affect Google Search rankings or AI Overviews.
Example robots.txt
This example allows AI search and user-requested fetching but blocks training crawlers:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlCrawlers not named in the file follow the rules under “User-agent: *”. There is no need to list every search crawler with an Allow rule if the default already allows them.
Things robots.txt cannot do
- It cannot enforce anything. Established companies follow it; unknown scrapers may ignore it.
- User-triggered fetchers are a grey area. Some companies, including Perplexity, state that their user-requested fetcher may not follow robots.txt, on the grounds that a person asked for the page.
- It does not remove pages from an index. A blocked URL can still be listed if other sites link to it.
- It cannot notify crawlers of changes. For faster indexing in Bing and some other engines, use IndexNow.
Check your CDN and firewall too
Many sites block AI crawlers without knowing it. Security plugins, hosting firewalls and CDNs can challenge or block unfamiliar bots. In July 2025 Cloudflare started blocking known AI crawlers by default for new domains, and site owners have to change the setting to allow them. If AI search crawlers never appear in your server logs, check these layers before blaming robots.txt. Our free AI crawler access checker tests robots.txt, firewall responses and rendering for ten crawlers in one command.
How to verify a crawler is genuine
Anyone can send a request that claims to be GPTBot or Googlebot. To trust log data:
- Check the requesting IP address against the ranges the company publishes (OpenAI, Anthropic, Perplexity and others publish them).
- For Googlebot and Bingbot, run a reverse DNS lookup on the IP and then a forward lookup on the hostname to confirm it resolves back to the same IP.
- Treat requests that fail these checks as ordinary bots, whatever their user agent says.
Frequently asked questions
If I block GPTBot, will ChatGPT stop citing me?
Not necessarily. ChatGPT search uses OAI-SearchBot’s index. Blocking GPTBot only affects training.
Does WordPress have a robots.txt file?
WordPress generates a virtual robots.txt if there is no physical file in the site root. Plugins and themes can add rules to it. If a physical robots.txt exists, it replaces the virtual one.