To get your website cited by Claude, allow Anthropic’s search crawler, Claude-SearchBot, and its user-triggered fetcher, Claude-User, to reach your pages, make sure the text is in the HTML, and write pages that answer specific questions directly with evidence. Claude can search the web and shows citations for the sources it uses, and it can also read a page when a user pastes a link. Anthropic documents three separate user agents, so a site can allow search and user visits while making its own decision about training.
Anthropic’s three user agents
| User agent | Purpose | Effect of blocking |
|---|---|---|
| Claude-SearchBot | Crawls to improve the quality of search results shown in Claude | Your pages may not be indexed for Claude’s search and may appear less in its answers |
| Claude-User | Fetches a page when a Claude user asks a question that needs it or shares a link | Claude cannot read your page on a user’s behalf |
| ClaudeBot | Collects web content that may be used to train Anthropic’s models | Content collected after the block is excluded from training data; search is controlled separately |
Anthropic’s documentation says its bots respect robots.txt. Decide on each separately; our AI crawler decision guide covers the choice.
Step 1: allow access
- Check robots.txt for rules that name Claude-SearchBot, Claude-User or ClaudeBot, and for broad rules that block all bots.
- Check your CDN and firewall; Cloudflare’s AI crawler settings and many security plugins block Anthropic’s crawlers by default. See stopping Cloudflare from blocking AI crawlers.
- Test with our AI crawler access checker, which includes Claude-SearchBot and ClaudeBot.
Step 2: make pages readable without JavaScript
Like most AI crawlers, Anthropic’s fetchers read the HTML your server sends. Content added by JavaScript after the page loads may never be seen. Our render test shows how large the difference can be.
Step 3: write for the people who use Claude
Claude is widely used for work: research, analysis, writing and software development. Questions tend to be detailed and specific. Pages that do well give precise, well-structured answers with sources, such as documentation, technical guides, methodology pages, pricing details and comparison tables. Put the answer in the first sentence of each section, name the subject, and include the numbers and conditions that make it true. See 10 before-and-after rewrites.
Step 4: make shared links work well
When a user pastes your URL into Claude and asks about it, Claude-User fetches the page. If the page is blocked, behind a cookie wall, or empty without JavaScript, Claude cannot help the user with it, and the user may form their view from other sources. A clean, server-rendered page with a clear summary at the top serves these requests best.
Step 5: strengthen search visibility and consistency
- Pages that rank well in web search for the sub-questions of a buyer’s question are more likely to be found by any assistant.
- Consistent company details across your site, profiles and directories help Claude describe you correctly. See entity resolution.
Step 6: measure
- Visits from claude.ai appear as referrals in analytics; include them in your AI assistants channel group.
- Check your logs for Claude-SearchBot and Claude-User requests and their status codes.
- Test your buyer questions in Claude with web search enabled, several times a month.
Frequently asked questions
Does blocking ClaudeBot remove us from Claude’s answers?
ClaudeBot relates to training data. Search visibility depends on Claude-SearchBot and user fetches on Claude-User, which are controlled separately.
Does Claude use llms.txt?
Anthropic has not said that Claude uses llms.txt to choose search sources. Some developer documentation sites publish it for coding tools, which is a reasonable low-cost addition for documentation.
Is Claude important for B2B companies?
For companies whose buyers are professionals, analysts or developers, it can be. Test your own buyer questions to see whether Claude cites your competitors.
Sources
- Anthropic, documentation on ClaudeBot, Claude-User and Claude-SearchBot
- Anthropic, announcement of web search in Claude, 2025