Most robots.txt files written in the last two years contain a block of AI crawler rules copied from a blog post. A good share of them block the wrong bots: they disallow the crawler that would have cited the site in ChatGPT or Perplexity while leaving the training crawler alone, which is the exact opposite of what the person intended.
The reason is that these crawlers do three unrelated jobs, operators use similar names for different purposes, and the lists circulating online are copied from each other rather than from documentation. This page is built only from each operator’s own published documentation, read on 2 November 2026, with the source for each named at the end. Where a vendor does not state something, it says so rather than guessing.
If you want the decision rather than the reference, our position on whether to block at all is in should Indian businesses block GPTBot, and the concept page is AI crawlers and robots.txt.
The distinction that decides everything
Three jobs, and conflating them is the whole problem.
Training crawlers collect content that may be used to train models. Blocking one is a statement about future training data. It does not remove you from anything users see today, and it does not take effect retroactively.
Search and retrieval crawlers build the index an assistant draws on when answering. Blocking one of these removes you from that assistant’s answers and citations. This is the one people block by accident, and it is the most expensive mistake in the list.
User-initiated fetchers fetch a page because a person asked the assistant about it. Several operators state plainly that these do not follow robots.txt, because the request came from a user rather than from a crawl. You cannot reliably block these, and blocking the token does not stop the fetch.
So the useful question is never “should I block AI crawlers”. It is “do I want to be cited, and do I want to be trained on”, which have different answers for most businesses and different tokens.
OpenAI
OpenAI documents four crawlers, and the separation is the clearest of any operator.
GPTBot is training. The documentation says it crawls content that may be used to train OpenAI’s generative AI foundation models. Example token: GPTBot/1.4, with the version subject to change, identifying itself at openai.com/gptbot. Disallowing it signals that your content should not be used for training.
OAI-SearchBot is search, and this is the one you almost certainly want to allow. It surfaces websites in ChatGPT’s search features, and OpenAI states that opted-out sites will not appear in ChatGPT search answers, though they can still appear as navigational links. Example token: OAI-SearchBot/1.4. Changes to robots.txt take around 24 hours to take effect for search.
ChatGPT-User is user-initiated browsing: it visits pages when someone asks ChatGPT or a custom GPT a question, and it powers GPT Actions. OpenAI states it is not used for automatic crawling and does not determine whether content appears in Search, and that robots.txt rules may not apply because the visits are user-initiated. Example token: ChatGPT-User/1.0.
OAI-AdsBot checks that landing pages submitted as ChatGPT ads comply with policy, and may use landing page content to judge ad relevance. OpenAI states it only visits submitted ad pages and that its data is not used to train generative AI foundation models. Example token: OAI-AdsBot/1.0. The documentation does not state its robots.txt behaviour.
The common error here is a single rule disallowing everything beginning with GPT or OAI. That blocks OAI-SearchBot and removes the site from ChatGPT’s search answers, which for most businesses is the opposite of the goal. If you want to be cited but not trained on, allow OAI-SearchBot and disallow GPTBot.
Anthropic
Three crawlers, with the same three-way split.
ClaudeBot collects web content for model training. Anthropic’s documentation says it “helps enhance the utility and safety of our generative AI models”, and that disabling it signals future material should be excluded from training datasets.
Claude-SearchBot indexes content to improve search result quality, in Anthropic’s words it “navigates the web to improve search result quality for users”, and disabling it may reduce visibility and accuracy in user search results.
Claude-User retrieves pages when a Claude user asks a question that requires them. Disabling it stops retrieval for user queries, which Anthropic notes may reduce visibility for user-directed search.
Usefully, Anthropic states that its bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt”, for all three, and that it supports what it calls “the non-standard Crawl-delay extension to robots.txt”. Note that Anthropic publishes the tokens rather than full user-agent strings, and that the robots.txt rule must be in each subdomain’s top-level directory.
Perplexity
Two crawlers, and the documentation is explicit about what each is not for.
PerplexityBot exists to show and link websites in Perplexity search results, and the documentation states it is “not used to crawl content for AI foundation models”. It respects robots.txt and Perplexity recommends allowing it. Example string: Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot). Published IP ranges are available as JSON at perplexity.ai/perplexitybot.json.
Perplexity-User fetches a page when a user asks about it. The documentation says it is “not used for web crawling or to collect content for training AI foundation models”, and that because a user requested the fetch, “this fetcher generally ignores robots.txt rules”. Its IP ranges are published at perplexity.ai/perplexity-user.json.
Since neither is a training crawler by Perplexity’s own account, blocking PerplexityBot buys you nothing on the training question and costs you citations. If you want to be in Perplexity’s answers, allow it and verify it by IP using the published ranges, which is the approach in verifying Googlebot and GPTBot in server logs.
Google’s arrangement is different and is the one most often misunderstood, because the AI control is not a crawler at all.
Google-Extended is a control token, not a crawler. Google states it “has no separate user agent string” and that “the robots.txt user-agent token is used in a control capacity”. It governs whether crawled content can be used to train future Gemini models and to support grounding in Gemini Apps and Vertex AI. Critically, Google states that “Google-Extended does not impact a site’s inclusion in Google Search”.
That last sentence is the important one. Disallowing Google-Extended is the only clean way to opt out of Gemini training without touching Search, and it costs nothing in Search. There is no equivalent lever that removes you from AI Overviews while keeping you in Search, because AI Overviews draw on the Search index that Googlebot builds.
Googlebot is the main Search crawler, covering Discover, and its rules also affect Images, Video and News. Google states its common crawlers “always obey robots.txt rules when crawling automatically”. Blocking Googlebot removes you from Google. Nobody should need telling, and it happens anyway, usually via a staging robots.txt that went live.
GoogleOther is a generic crawler used by various product teams for publicly accessible content including internal research and development, and Google states it does not affect any specific product. Google-CloudVertexBot performs crawls that site owners themselves request in order to build Vertex AI Agents, and Google states “It has no effect on Google Search or other products”.
Apple
The same control-token pattern as Google, and worth knowing because Apple’s surfaces are heavily used in India on iPhones.
Applebot powers search features in Spotlight, Siri and Safari, and Apple states the crawled data “may also be used to help train Apple foundation models” and may supply context for AI-generated answers. It “respects standard robots.txt directives in general search crawls that are targeted at Applebot”. Two quirks worth noting: if your robots.txt has no Applebot rules but does have Googlebot rules, Applebot follows the Googlebot rules, and Applebot ignores crawl-delay.
Applebot-Extended is the training opt-out and does not crawl: Apple states plainly “Applebot-Extended does not crawl webpages”. Disallowing it opts you out of training Apple’s general-purpose models, and Apple confirms “Webpages that disallow Applebot-Extended can still be included in search results” and that “Site rules for Applebot-Extended are not considered in ranking for Search”.
That Googlebot inheritance rule catches people out. A site with a restrictive Googlebot block and no Applebot rules has, without intending to, applied the same restriction to Apple’s crawler.
Meta
Five crawlers, and Meta’s documentation separates them by purpose.
meta-externalagent is the training one: Meta’s documentation ties it to “training foundation AI models or improving products by indexing content directly”.
meta-webindexer crawls to improve Meta AI search result quality, and Meta says allowing it “helps us cite and link to your content”, which makes it the citation crawler rather than a training one.
meta-externalfetcher fetches individual links when a user requests it, supporting agentic features, and Meta states plainly “this crawler may bypass robots.txt rules”.
facebookexternalhit fetches content shared on Meta’s apps and gathers, caches and displays the title, description and thumbnail. It is not a training crawler, and Meta notes it might bypass robots.txt when performing security or integrity checks. Blocking this one breaks your link previews on Facebook, WhatsApp and Instagram, which is a marketing cost people rarely connect to the robots.txt rule that caused it.
meta-externalads crawls for advertising and business products.
Meta notes that robots.txt changes may take up to 24 hours to apply because its crawlers may cache robots.txt for that long.
Common Crawl
CCBot deserves its own mention because its output feeds a great many models indirectly. It is, in Common Crawl’s description, a “Nutch-based web crawler that makes use of the Apache Hadoop project”, with the full string CCBot/2.0 (https://commoncrawl.org/faq/), producing and maintaining an open repository of web crawl data. The page documents blocking it with a User-agent: CCBot and Disallow: / rule, which implies it honours the directive, though the page does not state that in those words, and it does not mention crawl-delay.
Blocking CCBot is the one decision with the widest downstream effect, because the archive is used by researchers and by many model builders who never crawl your site themselves. It is also the one with no visibility upside at all, since nothing cites you from it.
The reference table
All from each operator’s own documentation, read 2 November 2026.
| Token | Operator | Job | Honours robots.txt | Blocking it costs you |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | Yes, as a training signal | Nothing user-facing |
| OAI-SearchBot | OpenAI | ChatGPT search index | Yes, about 24 hours to apply | Appearance in ChatGPT search answers |
| ChatGPT-User | OpenAI | User-initiated fetch | May not apply | Not reliably blockable |
| OAI-AdsBot | OpenAI | Ad landing page checks | Not stated | Ad policy checks on your pages |
| ClaudeBot | Anthropic | Training | Yes | Nothing user-facing |
| Claude-SearchBot | Anthropic | Search index | Yes | Visibility in Claude search results |
| Claude-User | Anthropic | User-initiated fetch | Yes | Retrieval for user queries |
| PerplexityBot | Perplexity | Search index, not training | Yes | Citations in Perplexity |
| Perplexity-User | Perplexity | User-initiated fetch | Generally ignores it | Not reliably blockable |
| Google-Extended | Control token for Gemini training and grounding | Yes, control only | Nothing in Search, by Google’s statement | |
| Googlebot | Search index | Yes | Google Search entirely | |
| GoogleOther | Generic product and research crawls | Yes | No specific product | |
| Google-CloudVertexBot | Owner-requested Vertex AI Agent crawls | Yes | Nothing, unless you requested it | |
| Applebot | Apple | Spotlight, Siri, Safari search, may train Apple models | Yes, inherits Googlebot rules if absent, ignores crawl-delay | Apple search surfaces |
| Applebot-Extended | Apple | Control token for training opt-out, does not crawl | Yes, control only | Nothing in search or ranking, by Apple’s statement |
| meta-externalagent | Meta | Training and direct indexing | Presented as controllable | Nothing user-facing |
| meta-webindexer | Meta | Meta AI search quality and citation | Presented as controllable | Citations in Meta AI |
| meta-externalfetcher | Meta | User-requested agentic fetch | May bypass it | Not reliably blockable |
| facebookexternalhit | Meta | Link previews | May bypass for integrity checks | Link previews on Facebook, WhatsApp, Instagram |
| meta-externalads | Meta | Advertising products | Presented as controllable | Ad-related crawling |
| CCBot | Common Crawl | Open crawl archive | Blocking documented | Nothing, no citation upside either |
The four mistakes we see most
Wildcard blocks that catch the search crawlers. A rule aimed at training bots that also disallows OAI-SearchBot, Claude-SearchBot, PerplexityBot or meta-webindexer. The site then disappears from AI answers while its content continues to be trained on by anyone who used Common Crawl. Worst of both outcomes.
Blocking Googlebot to control AI Overviews. There is no separate lever. Google-Extended covers Gemini training and grounding, not AI Overviews, and AI Overviews use the Search index. Blocking Googlebot removes you from Google.
Expecting robots.txt to stop user-initiated fetchers. OpenAI, Perplexity and Meta each document that their user-triggered fetchers may or do ignore robots.txt. If a page must not be fetched on request, robots.txt is not the mechanism; authentication is.
Rules that never reach the crawler. The most common failure is not the rule at all. A content delivery network or bot protection layer blocks the crawler before robots.txt is ever read, so an allow rule achieves nothing. On Cloudflare in particular this is routine, and the fix is in how to stop Cloudflare from blocking AI crawlers. Verify with the AI crawler access checker rather than assuming.
A sensible configuration for most Indian businesses
Not advice for every case, but the default we would start from for a business that wants customers to find it.
Allow every search and retrieval crawler: OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer, Googlebot, Applebot. These are how you appear in answers, and there is no commercial argument for blocking them unless you are a paywalled publisher.
Decide training separately and deliberately: GPTBot, ClaudeBot, meta-externalagent, Google-Extended, Applebot-Extended, CCBot. For most service businesses, manufacturers and local businesses, being in training data is harmless and arguably useful. For publishers whose content is the product, the calculation is different and the decision guide applies.
Never block facebookexternalhit unless you want your own shared links to render without a preview.
Then verify from the logs rather than from the file, because the only proof a crawler reached you is that it did. The log analysis approach is in log file analysis at scale with Python, and crawler identity verification in verifying Googlebot and GPTBot in server logs.
What this page deliberately does not do
It does not list every AI crawler in existence. Tokens appear and disappear monthly, several operators publish nothing, and a list padded with unverifiable names is the problem this page exists to fix. Amazonbot, Bytespider, Diffbot, Omgilibot and others crawl, and we have not included them because we could not verify their current documented purpose and robots.txt behaviour from their own sources on the date above. If a crawler matters to your logs and is not here, the test is whether its operator documents it.
It also does not tell you that blocking AI crawlers protects your content. It does not. Blocking is a forward-looking signal to operators who honour it, with no retroactive effect and no bearing on anyone who ignores it.
If your robots.txt was written from a blog post and nobody has checked what it actually blocks, get in touch.
Last verified: 2 November 2026. We review this page monthly, because this is the fastest-changing reference on this site.
Sources, all read 2 November 2026. OpenAI, “Bots” documentation at developers.openai.com. Anthropic, “Does Anthropic crawl data from the web” support article at support.claude.com. Perplexity, “Bots” guide at docs.perplexity.ai. Google Search Central, “Google crawlers and fetchers” at developers.google.com. Apple Support, “About Applebot” article 119829. Meta for Developers, sharing bot documentation at developers.facebook.com. Common Crawl, CCBot page at commoncrawl.org.