How to Stop Cloudflare From Blocking AI Crawlers (Without Turning Off Protection)

Five Cloudflare settings that can block ChatGPT, Claude and Perplexity crawlers even when robots.txt allows them, and how to allow AI search while keeping protection on.

Ashruba Pankhade Published 4 min read

If your site uses Cloudflare, AI crawlers may be blocked even when your robots.txt allows them. Since July 2025, Cloudflare blocks known AI crawlers by default for newly added domains, and several other Cloudflare features can block or rewrite access for bots. To allow AI search crawlers, check five places: the AI bot blocking setting, the per-crawler controls, managed robots.txt, bot protection modes and your custom WAF rules. Then confirm the result with a test and with Cloudflare’s own security logs.

Cloudflare renames and moves settings from time to time. The names below match its documentation and announcements from 2025; if a label differs in your dashboard, search the Cloudflare docs for the feature name.

Decide what you want first

Most businesses want AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) allowed, so they can appear in AI answers, and may want training crawlers (GPTBot, ClaudeBot, CCBot) blocked. Cloudflare’s controls can do either. Our decision guide for AI crawlers helps you choose.

1. The “Block AI bots” setting

In the Cloudflare dashboard, under your domain’s security or bot settings, there is a setting to block AI bots. It can apply to all pages, only to hostnames that show ads, or not at all. If it is on, known AI crawlers receive a block response regardless of robots.txt.

If you want AI search visibility, set it to not block, then control individual crawlers in the next step instead. Blocking everything is the simplest option but removes you from AI search answers too.

2. Per-crawler controls

Cloudflare’s AI crawler controls (introduced in 2024 as AI Audit and later developed into a crawl control section) list the AI crawlers visiting your site, how often they come, and let you allow or block each one. Use this to allow search crawlers and block training crawlers individually. It also shows which crawlers were blocked recently, which is the fastest way to see whether Cloudflare has been stopping them.

3. Managed robots.txt

Cloudflare can add rules to your robots.txt on your behalf, asking AI training crawlers not to use your content. If this is on, the robots.txt that visitors receive is different from the file on your server. Open https://www.yoursite.com/robots.txt in a browser and read what is actually served. Make sure the rules Cloudflare adds match your decision, and that they do not block search crawlers you want.

4. Bot protection modes

Bot Fight Mode and Super Bot Fight Mode challenge traffic that looks automated. Cloudflare treats verified crawlers differently from unknown bots, but crawlers that are not on its verified list, or that arrive from new IP addresses, can still be challenged. If you use these modes, check the bot settings for how “verified bots” and “definitely automated” traffic are handled, and watch the security events log for challenges served to AI crawlers.

5. Custom WAF rules

Look through your custom security rules and rate-limiting rules for conditions on user agents. Rules added years ago to stop scrapers, such as blocking anything containing “bot” or “crawler”, catch AI crawlers too. Add exceptions for the crawlers you want, or narrow the rules.

# Example: a rule expression that allows three AI search crawlers
# to skip a stricter custom rule (Cloudflare rule language)
(http.user_agent contains "OAI-SearchBot") or
(http.user_agent contains "Claude-SearchBot") or
(http.user_agent contains "PerplexityBot")

Matching on user agent alone lets anyone pretend to be those crawlers. Where Cloudflare offers verified-bot conditions, prefer those.

Confirm the result

  1. Run our AI crawler access checker on your key pages. Status 200 and full word counts for the search crawlers are what you want.
  2. Open Cloudflare’s security events and filter by user agent for OAI-SearchBot, PerplexityBot and Claude-SearchBot. Look for blocks or challenges after your change.
  3. Check your origin server logs over the following week for visits from these crawlers. Our guide to verifying real crawler visits in server logs includes a script for this.

Common mistakes

  • Allowing crawlers in robots.txt and assuming that is enough, while Cloudflare blocks them before robots.txt matters.
  • Turning off all bot protection to fix one crawler. Use per-crawler settings or narrow exceptions instead.
  • Forgetting that staging or old subdomains have their own settings.
  • Not re-checking after Cloudflare plan changes or new feature rollouts.

Frequently asked questions

Our domain was added before July 2025. Are we affected?

The default block applied to new domains, but older domains may have turned on AI bot blocking manually or through a setup wizard. Check the setting rather than assuming.

What is Cloudflare’s pay per crawl?

Cloudflare announced a system in 2025 that lets site owners charge AI crawlers for access. It was in private beta at launch. It mainly concerns publishers who want to license content.

Does allowing AI crawlers slow the site down?

Usually not noticeably. Crawl rates from AI search crawlers are modest for most business sites. The per-crawler view shows the actual request volume.

Sources

  1. Cloudflare, announcement on blocking AI crawlers by default, July 2025
  2. Cloudflare documentation on AI crawler controls, managed robots.txt and bot management
  3. OpenAI, “Overview of OpenAI crawlers”

Related reading