Is Cloudflare Blocking AI Crawlers on Your Website? How to Check, and What to Ask Your Developer

Your website opens fine and Google sends visitors. Yet when a buyer asks ChatGPT, Claude or Perplexity for a firm like yours, your name never comes up. One possible cause sits where few owners look: the security service in front of your website.

AI assistants read the web through crawlers: programs that fetch pages for a search engine or an assistant. Security services such as Cloudflare exist to stop unwanted bots. Set one way, they also turn away the crawlers that decide whether an assistant can describe you. People can still open your site, so nothing looks broken.

This article explains which settings matter, how to find out in about ten minutes, and what to ask your developer. I check this on every site I audit, and crawler and firewall checks open my AI Visibility Deep Audit. Most owners can do the first round themselves.

The short version

• Most B2B service sites are fine. Since 15 September 2026, new Cloudflare sites without ads start with every AI assistant allowed.
• The settings that cost you: Training on Block (since September, that also turns away Google and Bing), Search or Agent on Block, a bot-protection switch, or an old firewall rule.
• Saying no to AI training is a fair choice. Make it with the option called Disallow AI Training, never with Block.
• You can read your settings in ten minutes. A free outside test shows whether the crawlers actually get in.
• Ask your developer for proof: a test visit per crawler, not only a look at the settings.

How can a security setting keep AI assistants away?

It was built to stop bots, and AI assistants send bots. A security service sits in front of your website and sees every visit first. It judges each visitor by its name, where it comes from and how it behaves. It can refuse a visit before your site ever sees it.

AI assistants send three kinds of visitor:

• Search crawlers read your pages ahead of time, so an assistant can find and cite you later. OpenAI, Anthropic and Perplexity each run one. Google’s and Bing’s normal crawlers do this job for their AI answers too.
• Live fetchers open a page the moment someone asks about it, for example when a buyer pastes your address into ChatGPT.
• Training crawlers collect text to teach future models. Saying no to them leaves today’s answers almost untouched, if it is done with the right setting.

The first two decide whether you are named. A block on either is the expensive kind.

Cloudflare changed its AI settings three times in 2026, so this article uses its names. Other security services and some web hosts have bot protection too; the question is the same. Google’s guide to its AI features asks that crawling be allowed “in robots.txt, and by any CDN or hosting infrastructure”. robots.txt is a small public file that tells crawlers what they may read. A CDN is a delivery network in front of your site, such as Cloudflare.

What did Cloudflare change in 2026?

It replaced one “Block AI bots” switch with three separate choices. From 15 September 2026, new sites without ads start with all three allowed. In order:

• July 2025. Cloudflare made blocking AI crawlers the default for new domains. Sites added over the next year started with the block on, unless someone chose otherwise.
• 1 July 2026. Every plan, the free one included, got three separate settings: Search, Agent and Training. Each can be Allow, Block, or Block on pages with ads.
• 21 August 2026. Cloudflare announced Bot Preference Sync. It writes those choices into your robots.txt for you, above whatever your own file says, and it is on by default for new customers.
• 15 September 2026. A new Training option, Disallow AI Training, says no to training while Google, Bing and Apple keep crawling for search. Block under Training now also covers Google’s, Bing’s and Apple’s crawlers, which crawl for both search and training. The old switch was retired, and existing sites were moved to the new settings.

New sites are now asked about ads at sign-up. With ads, training is disallowed and live fetchers are blocked on ad pages.

Does this affect my business?

For most B2B service firms without ads, probably not. Four situations make it worth a look.

1. Your site shows ads. If you told Cloudflare so, live fetchers are blocked on pages it detects as showing ads. It only says the detection is automated, so check the pages that matter.
2. Your site joined Cloudflare between July 2025 and September 2026, or someone switched on the old “Block AI bots”. In September that became “no training” plus live fetchers blocked on ad pages. Without ads, nothing stops you being cited, but you have a training opt-out you may never have chosen.
3. Someone set Training to Block after 15 September 2026. This is the quiet one. It now turns away Google’s, Bing’s and Apple’s search crawlers too.
4. A firewall rule or the bot-protection switch is in the way. Both act on their own. Cloudflare’s documentation warns that an older firewall rule can still turn away a crawler you set to Allow.

Blocking search on purpose is rare: Cloudflare’s 15 September post says only a tiny share of its sites do it, far fewer than block training.

Say you run a dental clinic with two locations. Your agency put the site on Cloudflare in early 2026, old AI block on. Since September it reads “no training, live fetchers kept off ad pages”, and your site has no ads. In a case like this nothing needs repairing. Keeping the training opt-out is a business choice, covered below.

What could one wrong setting cost a firm like mine?

A quiet month or two that nobody connects to a setting. Here is how that can happen.

Picture a 12-person tax firm in Munich. In October, a partner reads that AI companies train on websites and asks the office manager to switch that off. She logs in to Cloudflare, finds Training, and picks Block. It looks like the obvious choice.

Nothing breaks, and the site opens for everyone. But since 15 September, Block under Training also turns away the crawlers behind Google and Bing. In a case like this, pages can start to drop out of search over the following weeks. With them go the firm’s chances in Google’s AI Overviews and in Microsoft Copilot’s answers.

Say the website brings the firm six new enquiries a month, most of them from search. Enquiries vary anyway, so a month with three draws no attention. Nobody sees an error message, and the setting was changed weeks earlier.

The repair takes about five minutes: switch Training to Disallow AI Training. That keeps the partner’s wish of no training and lets the search crawlers back in. Nobody here was careless. One word in a menu changed meaning in September, and only a check finds that.

Should I block AI training or allow it?

For most B2B service firms, I allow it. A model that has read your service pages can describe what you do even when it doesn’t search the web. That is my reasoning, not a measured effect.

If your content is your product, say no to training with Disallow AI Training. It has two small trade-offs, for Google’s Gemini app and for Bing; they are in the developer section.

My default for a B2B service firm: all three AI settings on Allow and the bot-protection switch off or checked. No firewall rules that pick on crawler names. It is the setup on my launch checklist for this site.

What I rejected, and why:

• Pay per crawl. This Cloudflare private beta charges crawlers for access. A price in front of the assistants works against a firm that wants to be cited.
• The old managed robots.txt, Cloudflare’s ready-made file. It wrote someone else’s preference into yours, and Cloudflare is retiring it anyway.
• Training on Block. It now takes Google and Bing with it. Disallow AI Training does the job precisely.

Who should not copy my default: publishers who live on ads or paywalls, firms with content licensing deals, and anyone with material a model may not keep.

How can I find out in a few minutes?

Two looks tell you most of it: a test from outside, and Cloudflare’s own record from inside.

From the outside. Every crawler gives a name when it knocks, called its user agent. My free scan reads your robots.txt rules for 10 AI crawlers and sends a test visit under each crawler’s name. If a search crawler or a live fetcher is turned away, you have found the problem.

The test has a limit. Real crawlers also arrive from known addresses, which a security service can treat differently. A clean result is a good sign, not full proof.

From the inside. Cloudflare keeps its own list of the AI crawlers that visited and whether they got in. On the free plan it covers the last 24 hours. If a crawler you allowed shows refused visits, something else is in the way, such as a firewall rule.

What I found on my own sites. On 5 October 2026 I ran the outside test against an older browser game of mine, on a subdomain of my old agency domain. Its robots.txt welcomes OpenAI’s and Anthropic’s training crawlers by name. Cloudflare refused both with “Your request was blocked”. Every search crawler and live fetcher got in, Google and Bing included. A product site of mine on another domain shows the same.

The file said yes and the security layer said no. Only the dashboard can show which setting does it: Disallow AI Training, which spares search, or Block, which doesn’t. That gap is why I test the response, not just the file.

What you can do this week

With rough times:

1. Find out who holds the login for Cloudflare or your web host’s security settings. Five minutes, or one email. If nobody knows, that is a finding in itself.
2. Run the free scan at the end of this article. About a minute.
3. Read the three AI settings: Search, Training and Agent. Ten minutes with the login.
4. If Training says Block or Block on pages with ads, switch it to Disallow AI Training, or to Allow if you are happy to be trained on. Two minutes.
5. Check whether the bot-protection switch is on (Cloudflare calls it Bot Fight Mode) and whether any firewall rules exist. Don’t delete a rule you don’t understand. Five minutes.
6. Send your developer the questions below. Five minutes.
7. Put a reminder in your calendar to look again in three months. One minute.

Questions to send your developer:

• Is our site behind Cloudflare or another security service, and who has the login?
• What do Search, Training and Agent say in the AI settings, and why?
• Is Bot Fight Mode on, and has it challenged any AI crawler recently?
• Do we have firewall rules that block by crawler name, country or “known bots”?
• Can you send me a test result for each AI crawler, and does our robots.txt say the same as the settings?

What have I seen on real sites?

The most useful thing I have built around this problem is an alarm, not a setting. For Corvane Legal, a German law firm, I built a content engine that now has 1,100+ pages live on the firm’s own domain. It ships with an allowlist for AI crawlers, because a site can start blocking them without anyone deciding to. It tracks 16 AI crawlers by name and sends an alert when their visits stop.

16✓ AI crawlers tracked by name observed 2026-10-05 · crawler module

That alert exists for the failure in this article. A setting changes or someone adds a firewall rule, and the first symptom is silence. Since the handover in Sept 2026, it runs in the firm’s own Cloudflare account.

Access is only half the job. If your pages only appear after code runs in the visitor’s browser, the crawler gets in and finds an empty page. My old site showed crawlers that don’t run that code, the main AI crawlers among them, 0 words of content. I wrote up how I found and fixed that.

For your web developer

Paths as of October 2026, Free plan.

• AI bot policies: Security > Settings, filter Bot traffic, Configure AI bot policies. Training on Block or Block on pages with ads now includes Googlebot, Bingbot and Applebot; use Disallow AI Training. If the legacy Block AI bots toggle still shows, confirm the migrated values. The Block AI Bots docs page still described the July plan on 5 October 2026; trust the 15 September post and the dashboard.
• Bot Fight Mode: Security > Settings > Bot traffic; its challenges show Service = Bot Fight Mode under Security > Analytics > Events. No WAF rule can skip it: switch it off, or on Pro and up use Super Bot Fight Mode with Verified bots on Allow.
• WAF custom rules (Security > Security rules > Custom rules) run before the AI Crawl Control rule. Look for block or challenge by user agent, country, ASN or known bots.
• AI Crawl Control > Crawlers: the Action column, and Metrics by crawler and status code.
• Live fetchers: OpenAI and Perplexity say robots.txt may not apply to ChatGPT-User and Perplexity-User; the Agent setting or a WAF rule decides.
• robots.txt: Preference Sync writes lines between BEGIN Cloudflare and END Cloudflare markers at the top. Add Content-Signal: search=yes, ai-input=yes, ai-train=yes (or ai-train=no) and repeat it in every named group you allow: a crawler obeys only the most specific group naming it. Search Console may flag it as “Syntax not understood”; Cloudflare reports no effect on crawling.
• Disallow AI Training trade-offs: its Google-Extended opt-out keeps Google Search and AI Overviews but drops grounding in Gemini Apps. Bing can’t read a no-training line until early 2027 (Microsoft’s target); Cloudflare points to the NOARCHIVE meta tag meanwhile.
• Prove it: the loop below only catches rules that match on user agent, which is how AI Crawl Control detects crawlers on Free. Real crawlers come from published IP ranges (OpenAI’s searchbot.json and chatgpt-user.json, Anthropic’s claude.com/crawling/bots.json, Perplexity’s perplexitybot.json and perplexity-user.json); compare origin logs with those.

for bot in OAI-SearchBot/1.4 ChatGPT-User/1.0 GPTBot/1.4 Claude-SearchBot/1.0 Claude-User/1.0 \
           ClaudeBot/1.0 PerplexityBot/1.0 Perplexity-User/1.0 Googlebot/2.1 bingbot/2.0; do
  printf '%-22s ' "$bot"
  curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; $bot)" https://example.com/
done

When should I hire someone?

When the checks don’t come back clean, or when you want to know whether the assistants actually name you.

If the free scan shows every search crawler and live fetcher getting in, and your settings read Allow or Disallow AI Training, you don’t need to pay me for this part. You are done, and my AI-ready website checklist covers what comes next.

If not, the Deep Audit costs €1,500$1,650 and takes 7 business days. It tests access for 10 AI crawlers, puts 30 questions to each of 4 assistants, and ends with a recorded 45-minute walkthrough with me. Fixed price. What you pay in cash is credited in full to any build within 60 days.

Start with the outside testThe free scan reads your robots.txt rules for the main AI crawlers and sends a test visit under each crawler's name.

Your website address Scan my site →