---
title: "Cloudflare Blocking AI Crawlers? A 10-Minute Check for Owners"
description: "A security setting at Cloudflare or your web host can quietly turn ChatGPT, Claude and Perplexity away. How to check in minutes, and what to ask your developer."
canonical: "https://webrevolution.si/insights/cloudflare-blocking-ai-crawlers/"
updated: "2026-11-08"
published: "2026-11-08"
words: 3363
jsKb: 4.7
---
[AI-ready websites](https://webrevolution.si/insights/topic/ai-ready-websites/)

# Is Cloudflare Blocking AI Crawlers on Your Website? How to Check, and What to Ask Your Developer

A security setting at Cloudflare or your web host can quietly turn ChatGPT, Claude and Perplexity away. How to check in minutes, and what to ask your developer.

[Vegard Wikeby](https://webrevolution.si/about/)

Written by Vegard with research agents. Every claim checked by me.

- Published 8 Nov 2026
- Updated 8 Nov 2026
- 10 min read
- Checked against 16 sources
- [See as AI](https://webrevolution.si/insights/cloudflare-blocking-ai-crawlers/index.md)

Short answer

Usually not, but ten minutes will tell you. Since 15 September 2026, new Cloudflare sites without ads start with AI assistants allowed. The settings that cost you: Training on Block (which now also turns away Google and Bing), Search or Agent on Block, a bot-protection switch, or an old firewall rule. Your site still works for people, so nobody notices. Check the settings yourself, then ask your developer to confirm.

[Need this done? AI Visibility Deep Audit](https://webrevolution.si/services/ai-visibility-audit/)

**Contents 11 sections**

Your website opens fine and Google sends visitors. Yet when a buyer asks ChatGPT, Claude or Perplexity for a firm like yours, your name never comes up. One possible cause sits where few owners look: the security service in front of your website.

AI assistants read the web through crawlers: programs that fetch pages for a search engine or an assistant. Security services such as Cloudflare exist to stop unwanted bots. Set one way, they also turn away the crawlers that decide whether an assistant can describe you. People can still open your site, so nothing looks broken.

This article explains which settings matter, how to find out in about ten minutes, and what to ask your developer. I check this on every site I audit, and crawler and firewall checks open my [AI Visibility Deep Audit](https://webrevolution.si/services/ai-visibility-audit/). Most owners can do the first round themselves.

## The short version

- Most B2B service sites are fine. Since 15 September 2026, new Cloudflare sites without ads start with every AI assistant allowed.
- The settings that cost you: Training on Block (since September, that also turns away Google and Bing), Search or Agent on Block, a bot-protection switch, or an old firewall rule.
- Saying no to AI training is a fair choice. Make it with the option called Disallow AI Training, never with Block.
- You can read your settings in ten minutes. A free outside test shows whether the crawlers actually get in.
- Ask your developer for proof: a test visit per crawler, not only a look at the settings.

## How can a security setting keep AI assistants away?

It was built to stop bots, and AI assistants send bots. A security service sits in front of your website and sees every visit first. It judges each visitor by its name, where it comes from and how it behaves. It can refuse a visit before your site ever sees it.

AI assistants send three kinds of visitor:

- **Search crawlers** read your pages ahead of time, so an assistant can find and cite you later. OpenAI, Anthropic and Perplexity each run one. Google’s and Bing’s normal crawlers do this job for their AI answers too.
- **Live fetchers** open a page the moment someone asks about it, for example when a buyer pastes your address into ChatGPT.
- **Training crawlers** collect text to teach future models. Saying no to them leaves today’s answers almost untouched, if it is done with the right setting.

The first two decide whether you are named. A block on either is the expensive kind.

Cloudflare changed its AI settings three times in 2026, so this article uses its names. Other security services and some web hosts have bot protection too; the question is the same. Google’s guide to its AI features asks that crawling be allowed “in robots.txt, and by any CDN or hosting infrastructure”. robots.txt is a small public file that tells crawlers what they may read. A CDN is a delivery network in front of your site, such as Cloudflare.

## What did Cloudflare change in 2026?

It replaced one “Block AI bots” switch with three separate choices. From 15 September 2026, new sites without ads start with all three allowed. In order:

- **July 2025.** Cloudflare made blocking AI crawlers the default for new domains. Sites added over the next year started with the block on, unless someone chose otherwise.
- **1 July 2026.** Every plan, the free one included, got three separate settings: Search, Agent and Training. Each can be Allow, Block, or Block on pages with ads.
- **21 August 2026.** Cloudflare announced Bot Preference Sync. It writes those choices into your robots.txt for you, above whatever your own file says, and it is on by default for new customers.
- **15 September 2026.** A new Training option, Disallow AI Training, says no to training while Google, Bing and Apple keep crawling for search. Block under Training now also covers Google’s, Bing’s and Apple’s crawlers, which crawl for both search and training. The old switch was retired, and existing sites were moved to the new settings.

New sites are now asked about ads at sign-up. With ads, training is disallowed and live fetchers are blocked on ad pages.

## Does this affect my business?

For most B2B service firms without ads, probably not. Four situations make it worth a look.

1. **Your site shows ads.** If you told Cloudflare so, live fetchers are blocked on pages it detects as showing ads. It only says the detection is automated, so check the pages that matter.
2. **Your site joined Cloudflare between July 2025 and September 2026, or someone switched on the old “Block AI bots”.** In September that became “no training” plus live fetchers blocked on ad pages. Without ads, nothing stops you being cited, but you have a training opt-out you may never have chosen.
3. **Someone set Training to Block after 15 September 2026.** This is the quiet one. It now turns away Google’s, Bing’s and Apple’s search crawlers too.
4. **A firewall rule or the bot-protection switch is in the way.** Both act on their own. Cloudflare’s documentation warns that an older firewall rule can still turn away a crawler you set to Allow.

Blocking search on purpose is rare: Cloudflare’s 15 September post says only a tiny share of its sites do it, far fewer than block training.

Say you run a dental clinic with two locations. Your agency put the site on Cloudflare in early 2026, old AI block on. Since September it reads “no training, live fetchers kept off ad pages”, and your site has no ads. In a case like this nothing needs repairing. Keeping the training opt-out is a business choice, covered below.

## What could one wrong setting cost a firm like mine?

A quiet month or two that nobody connects to a setting. Here is how that can happen.

Picture a 12-person tax firm in Munich. In October, a partner reads that AI companies train on websites and asks the office manager to switch that off. She logs in to Cloudflare, finds Training, and picks Block. It looks like the obvious choice.

Nothing breaks, and the site opens for everyone. But since 15 September, Block under Training also turns away the crawlers behind Google and Bing. In a case like this, pages can start to drop out of search over the following weeks. With them go the firm’s chances in Google’s AI Overviews and in Microsoft Copilot’s answers.

Say the website brings the firm six new enquiries a month, most of them from search. Enquiries vary anyway, so a month with three draws no attention. Nobody sees an error message, and the setting was changed weeks earlier.

The repair takes about five minutes: switch Training to Disallow AI Training. That keeps the partner’s wish of no training and lets the search crawlers back in. Nobody here was careless. One word in a menu changed meaning in September, and only a check finds that.

Found by AI · done for you

Google looks fine, yet ChatGPT, Claude and Perplexity never name you.

In the Deep Audit I test access for every crawler that matters, including Cloudflare and firewall traps, put real buyer questions to the assistants, and hand you a fixed-quote plan for what to fix first.

€1,500 · $1,650 7 business days

[Start with this →](https://webrevolution.si/start/?goal=audit&ref=cloudflare-blocking-ai-crawlers)

[See what's included](https://webrevolution.si/services/ai-visibility-audit/)

## Should I block AI training or allow it?

For most B2B service firms, I allow it. A model that has read your service pages can describe what you do even when it doesn’t search the web. That is my reasoning, not a measured effect.

If your content is your product, say no to training with Disallow AI Training. It has two small trade-offs, for Google’s Gemini app and for Bing; they are in the developer section.

My default for a B2B service firm: all three AI settings on Allow and the bot-protection switch off or checked. No firewall rules that pick on crawler names. It is the setup on my launch checklist for this site.

What I rejected, and why:

- **Pay per crawl.** This Cloudflare private beta charges crawlers for access. A price in front of the assistants works against a firm that wants to be cited.
- **The old managed robots.txt,** Cloudflare’s ready-made file. It wrote someone else’s preference into yours, and Cloudflare is retiring it anyway.
- **Training on Block.** It now takes Google and Bing with it. Disallow AI Training does the job precisely.

Who should not copy my default: publishers who live on ads or paywalls, firms with content licensing deals, and anyone with material a model may not keep.

## How can I find out in a few minutes?

Two looks tell you most of it: a test from outside, and Cloudflare’s own record from inside.

**From the outside.** Every crawler gives a name when it knocks, called its user agent. My free scan reads your robots.txt rules for 10 AI crawlers and sends a test visit under each crawler’s name. If a search crawler or a live fetcher is turned away, you have found the problem.

The test has a limit. Real crawlers also arrive from known addresses, which a security service can treat differently. A clean result is a good sign, not full proof.

**From the inside.** Cloudflare keeps its own list of the AI crawlers that visited and whether they got in. On the free plan it covers the last 24 hours. If a crawler you allowed shows refused visits, something else is in the way, such as a firewall rule.

**What I found on my own sites.** On 5 October 2026 I ran the outside test against an older browser game of mine, on a subdomain of my old agency domain. Its robots.txt welcomes OpenAI’s and Anthropic’s training crawlers by name. Cloudflare refused both with “Your request was blocked”. Every search crawler and live fetcher got in, Google and Bing included. A product site of mine on another domain shows the same.

The file said yes and the security layer said no. Only the dashboard can show which setting does it: Disallow AI Training, which spares search, or Block, which doesn’t. That gap is why I test the response, not just the file.

## What you can do this week

With rough times:

1. **Find out who holds the login** for Cloudflare or your web host’s security settings. Five minutes, or one email. If nobody knows, that is a finding in itself.
2. **Run the free scan** at the end of this article. About a minute.
3. **Read the three AI settings**: Search, Training and Agent. Ten minutes with the login.
4. **If Training says Block** or Block on pages with ads, switch it to Disallow AI Training, or to Allow if you are happy to be trained on. Two minutes.
5. **Check whether the bot-protection switch is on** (Cloudflare calls it Bot Fight Mode) and whether any firewall rules exist. Don’t delete a rule you don’t understand. Five minutes.
6. **Send your developer the questions below.** Five minutes.
7. **Put a reminder in your calendar** to look again in three months. One minute.

Questions to send your developer:

- Is our site behind Cloudflare or another security service, and who has the login?
- What do Search, Training and Agent say in the AI settings, and why?
- Is Bot Fight Mode on, and has it challenged any AI crawler recently?
- Do we have firewall rules that block by crawler name, country or “known bots”?
- Can you send me a test result for each AI crawler, and does our robots.txt say the same as the settings?

## What have I seen on real sites?

The most useful thing I have built around this problem is an alarm, not a setting. For Corvane Legal, a German law firm, I built a [content engine](https://webrevolution.si/work/law-firm-ai-content-engine/) that now has 1,100+ pages live on the firm’s own domain. It ships with an allowlist for AI crawlers, because a site can start blocking them without anyone deciding to. It tracks 16 AI crawlers by name and sends an alert when their visits stop.

[16 AI crawlers tracked by name observed 2026-10-05 · crawler module](https://webrevolution.si/work/law-firm-ai-content-engine/)

That alert exists for the failure in this article. A setting changes or someone adds a firewall rule, and the first symptom is silence. Since the handover in Sept 2026, it runs in the firm’s own Cloudflare account.

Access is only half the job. If your pages only appear after code runs in the visitor’s browser, the crawler gets in and finds an empty page. My old site showed crawlers that don’t run that code, the main AI crawlers among them, 0 words of content. I wrote up [how I found and fixed that](https://webrevolution.si/insights/website-invisible-to-chatgpt/).

## For your web developer

Paths as of October 2026, Free plan.

- **AI bot policies:** Security > Settings, filter Bot traffic, Configure AI bot policies. Training on Block or Block on pages with ads now includes Googlebot, Bingbot and Applebot; use Disallow AI Training. If the legacy Block AI bots toggle still shows, confirm the migrated values. The Block AI Bots docs page still described the July plan on 5 October 2026; trust the 15 September post and the dashboard.
- **Bot Fight Mode:** Security > Settings > Bot traffic; its challenges show Service = Bot Fight Mode under Security > Analytics > Events. No WAF rule can skip it: switch it off, or on Pro and up use Super Bot Fight Mode with Verified bots on Allow.
- **WAF custom rules** (Security > Security rules > Custom rules) run before the AI Crawl Control rule. Look for block or challenge by user agent, country, ASN or known bots.
- **AI Crawl Control > Crawlers:** the Action column, and Metrics by crawler and status code.
- **Live fetchers:** OpenAI and Perplexity say robots.txt may not apply to ChatGPT-User and Perplexity-User; the Agent setting or a WAF rule decides.
- **robots.txt:** Preference Sync writes lines between BEGIN Cloudflare and END Cloudflare markers at the top. Add `Content-Signal: search=yes, ai-input=yes, ai-train=yes` (or `ai-train=no`) and repeat it in every named group you allow: a crawler obeys only the most specific group naming it. Search Console may flag it as “Syntax not understood”; Cloudflare reports no effect on crawling.
- **Disallow AI Training trade-offs:** its Google-Extended opt-out keeps Google Search and AI Overviews but drops grounding in Gemini Apps. Bing can’t read a no-training line until early 2027 (Microsoft’s target); Cloudflare points to the NOARCHIVE meta tag meanwhile.
- **Prove it:** the loop below only catches rules that match on user agent, which is how AI Crawl Control detects crawlers on Free. Real crawlers come from published IP ranges (OpenAI’s `searchbot.json` and `chatgpt-user.json`, Anthropic’s `claude.com/crawling/bots.json`, Perplexity’s `perplexitybot.json` and `perplexity-user.json`); compare origin logs with those.

```bash
for bot in OAI-SearchBot/1.4 ChatGPT-User/1.0 GPTBot/1.4 Claude-SearchBot/1.0 Claude-User/1.0 \
           ClaudeBot/1.0 PerplexityBot/1.0 Perplexity-User/1.0 Googlebot/2.1 bingbot/2.0; do
  printf '%-22s ' "$bot"
  curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; $bot)" https://example.com/
done
```

## When should I hire someone?

When the checks don’t come back clean, or when you want to know whether the assistants actually name you.

If the free scan shows every search crawler and live fetcher getting in, and your settings read Allow or Disallow AI Training, you don’t need to pay me for this part. You are done, and my [AI-ready website checklist](https://webrevolution.si/insights/what-is-an-ai-ready-website/) covers what comes next.

If not, the Deep Audit costs €1,500 · $1,650 and takes 7 business days. It tests access for 10 AI crawlers, puts 30 questions to each of 4 assistants, and ends with a recorded 45-minute walkthrough with me. Fixed price. What you pay in cash is credited in full to any build within 60 days.

*Form · GET /audit/ · fields: Your website address · button: “Scan my site →”*

**Start with the outside test** The free scan reads your robots.txt rules for the main AI crawlers and sends a test visit under each crawler's name.

## Sources

Every factual claim above traces to one of these. Numbers about my own work come from my dated receipts.

1. [Cloudflare Blog: Have it both ways: stay discoverable in search while disallowing AI training (15 September 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) blog.cloudflare.com
2. [Cloudflare Blog: Say it once: introducing Bot Preference Sync (21 August 2026)](https://blog.cloudflare.com/bot-preference-sync/) blog.cloudflare.com
3. [Cloudflare Blog: Your site, your rules: new AI traffic options for all customers (1 July 2026)](https://blog.cloudflare.com/content-independence-day-ai-options/) blog.cloudflare.com
4. [Cloudflare Blog: Content Independence Day: no AI crawl without compensation (1 July 2025)](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/) blog.cloudflare.com
5. [Cloudflare Changelog: New options to manage AI traffic (1 July 2026)](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/) developers.cloudflare.com
6. [Cloudflare Docs: Block AI Bots](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) developers.cloudflare.com
7. [Cloudflare Docs: robots.txt setting (managed robots.txt and Content Signals)](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/) developers.cloudflare.com
8. [Cloudflare Docs: AI Crawl Control with Cloudflare WAF](https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/) developers.cloudflare.com
9. [Cloudflare Docs: AI Crawl Control, get started](https://developers.cloudflare.com/ai-crawl-control/get-started/) developers.cloudflare.com
10. [Cloudflare Docs: Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/) developers.cloudflare.com
11. [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots) developers.openai.com
12. [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518) support.claude.com
13. [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) docs.perplexity.ai
14. [Google: Google's common crawlers (Google-Extended)](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) developers.google.com
15. [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) developers.google.com
16. [Bing Webmaster Blog: Introducing AI Performance in Bing Webmaster Tools (10 February 2026)](https://blogs.bing.com/webmaster/2026/2/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview/) blogs.bing.com

## Changelog

- 8 Nov 2026 First published.

Reviewed by Vegard Wikeby on 8 Nov 2026. The Updated date changes only when the substance changes, never for a typo.

![Vegard Wikeby](https://webrevolution.si/_astro/vegard-headshot.CXuTy1_D_ZsxVMd.webp)

About the author

[Vegard Wikeby](https://webrevolution.si/about/)

I shipped my first e-commerce shop in 2006 and have worked independently since 2014. Today I direct a crew of AI agents that research, draft and test, and I decide what ships. This article was made the same way: agents gathered the sources, I checked every claim against them.

[Book 20 minutes](https://webrevolution.si/contact/#book)

[Write to me](https://webrevolution.si/start/?ref=cloudflare-blocking-ai-crawlers)

Personal reply within one business day. Agents propose. A human approves.

[**Vegard Wikeby** One man. An army of agents. I write these myself, with research agents.](https://webrevolution.si/about/)

Related service

[AI Visibility Deep Audit](https://webrevolution.si/services/ai-visibility-audit/)

A fixed-price audit of how AI crawlers, assistants and buyers see your site, with every fix ranked by effort and impact.

€1,500 · $1,650 7 business days

[Start with this](https://webrevolution.si/start/?goal=audit&ref=cloudflare-blocking-ai-crawlers)

See what AI sees on your site

Free AI-readiness audit. About 60 seconds.

*Form · GET /audit/ · fields: Your website address · button: “Scan my site →”*

Keep reading

## More on AI-ready websites

[AI-ready websites](https://webrevolution.si/insights/topic/ai-ready-websites/) 8 Nov 2026 · 11 min read · 20 sources

### [My Website Was Invisible to ChatGPT: It Read 0 Words. How to Check Yours, and What I Changed](https://webrevolution.si/insights/website-invisible-to-chatgpt/)

My old website showed AI assistants 0 words. What that means for your firm, how to check your own site in two minutes, and what fixed mine.

[AI-ready websites](https://webrevolution.si/insights/topic/ai-ready-websites/) 8 Nov 2026 · 11 min read · 23 sources

### [What Is an AI-Ready Website? What It Means for Your Business and What to Fix First](https://webrevolution.si/insights/what-is-an-ai-ready-website/)

What an AI-ready website means for a B2B firm, how to tell in an afternoon whether AI assistants can find and quote yours, and what to fix first.

Super Intelligence (SI) for business · monthly

## One email a month. What changed in AI, and what to do about it.

The month’s new articles plus a short, sourced digest of what moved in AI for B2B businesses. Researched by agents, approved line by line by me before it is sent. [See a sample issue](https://webrevolution.si/briefing/)

*Form · POST /api/briefing/subscribe · fields: Your work email · button: “Get the SI Briefing”*

One email a month. Unsubscribe in one click. You confirm by email first.

---

## Structured facts

The page's JSON-LD (schema.org), flattened to text.

### WebSite: WebRevolution.si
- url: https://webrevolution.si/
- inLanguage: en
- publisher: WebRevolution.si

### OnlineBusiness: WebRevolution.si
- alternateName: WebRevolution; webrevolution.si; WebRevolution SI
- url: https://webrevolution.si/
- logo: https://webrevolution.si/icon-512.png
- slogan: One man. An army of agents.
- description: One-man AI agency for B2B: AI-ready websites, AI content & distribution engines, AI agents and custom tools that AI assistants can find, cite and recommend.
- founder: Vegard Wikeby
- areaServed: Worldwide
- knowsAbout: Generative engine optimization; AI-ready websites; AI agents for business; AI content engines; Cloudflare Workers; Conversion rate optimization; Model Context Protocol
- contactPoint: ContactPoint (contactType: sales; url: https://webrevolution.si/contact/; availableLanguage: en)
- sameAs: https://www.linkedin.com/in/vegardwikeby; https://x.com/vegardwikeby

### Person: Vegard Wikeby
- url: https://webrevolution.si/about/
- jobTitle: Founder, WebRevolution.si
- worksFor: WebRevolution.si
- image: https://webrevolution.si/img/vegard-wikeby.jpg
- sameAs: https://www.linkedin.com/in/vegardwikeby; https://x.com/vegardwikeby
- knowsAbout: Web development; AI agents; Generative engine optimization; WordPress; E-commerce; Cloudflare

### BreadcrumbList
- Home › Insights › AI-ready websites › Is Cloudflare Blocking AI Crawlers on Your Website? How to Check, and What to Ask Your Developer

### BlogPosting: Is Cloudflare Blocking AI Crawlers on Your Website? How to Check, and What to Ask Your Developer
- description: A security setting at Cloudflare or your web host can quietly turn ChatGPT, Claude and Perplexity away. How to check in minutes, and what to ask your developer.
- url: https://webrevolution.si/insights/cloudflare-blocking-ai-crawlers/
- mainEntityOfPage: https://webrevolution.si/insights/cloudflare-blocking-ai-crawlers/
- image: https://webrevolution.si/og/insights/cloudflare-blocking-ai-crawlers.png
- inLanguage: en
- keywords: Cloudflare blocking AI crawlers
- articleSection: AI-ready websites
- wordCount: 2538
- datePublished: 2026-11-08
- dateModified: 2026-11-08
- author: Vegard Wikeby
- publisher: WebRevolution.si
- isPartOf: CollectionPage (name: AI-ready websites: readable by crawlers, usable by agents; url: https://webrevolution.si/insights/topic/ai-ready-websites/)
- citation:
  - CreativeWork (name: Cloudflare Blog: Have it both ways: stay discoverable in search while disallowing AI training (15 September 2026); url: https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/)
  - CreativeWork (name: Cloudflare Blog: Say it once: introducing Bot Preference Sync (21 August 2026); url: https://blog.cloudflare.com/bot-preference-sync/)
  - CreativeWork (name: Cloudflare Blog: Your site, your rules: new AI traffic options for all customers (1 July 2026); url: https://blog.cloudflare.com/content-independence-day-ai-options/)
  - CreativeWork (name: Cloudflare Blog: Content Independence Day: no AI crawl without compensation (1 July 2025); url: https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/)
  - CreativeWork (name: Cloudflare Changelog: New options to manage AI traffic (1 July 2026); url: https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/)
  - CreativeWork (name: Cloudflare Docs: Block AI Bots; url: https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)
  - CreativeWork (name: Cloudflare Docs: robots.txt setting (managed robots.txt and Content Signals); url: https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)
  - CreativeWork (name: Cloudflare Docs: AI Crawl Control with Cloudflare WAF; url: https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-waf/)
  - CreativeWork (name: Cloudflare Docs: AI Crawl Control, get started; url: https://developers.cloudflare.com/ai-crawl-control/get-started/)
  - CreativeWork (name: Cloudflare Docs: Bot Fight Mode; url: https://developers.cloudflare.com/bots/get-started/bot-fight-mode/)
  - CreativeWork (name: OpenAI: Overview of OpenAI crawlers; url: https://developers.openai.com/api/docs/bots)
  - CreativeWork (name: Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?; url: https://support.claude.com/en/articles/8896518)
  - CreativeWork (name: Perplexity: Perplexity crawlers; url: https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
  - CreativeWork (name: Google: Google's common crawlers (Google-Extended); url: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
  - CreativeWork (name: Google Search Central: AI features and your website; url: https://developers.google.com/search/docs/appearance/ai-features)
  - CreativeWork (name: Bing Webmaster Blog: Introducing AI Performance in Bing Webmaster Tools (10 February 2026); url: https://blogs.bing.com/webmaster/2026/2/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview/)