--- title: "Website Invisible to ChatGPT? How to Check Yours, and Fix It" description: "My old website showed AI assistants 0 words. What that means for your firm, how to check your own site in two minutes, and what fixed mine." canonical: "https://webrevolution.si/insights/website-invisible-to-chatgpt/" updated: "2026-11-08" published: "2026-11-08" words: 3485 jsKb: 4.7 --- [AI-ready websites](https://webrevolution.si/insights/topic/ai-ready-websites/) # My Website Was Invisible to ChatGPT: It Read 0 Words. How to Check Yours, and What I Changed My old website showed AI assistants 0 words. What that means for your firm, how to check your own site in two minutes, and what fixed mine. [Vegard Wikeby](https://webrevolution.si/about/) Written by Vegard with research agents. Every claim checked by me. - Published 8 Nov 2026 - Updated 8 Nov 2026 - 11 min read - Checked against 20 sources - [See as AI](https://webrevolution.si/insights/website-invisible-to-chatgpt/index.md) Short answer My old website looked complete to people, but the programs that read websites for ChatGPT, Claude and Perplexity received 0 words of it. The site was built inside each visitor's browser, and those programs skip that step: they saw a name and a one-line description, nothing about what I sold. The fix is a site where every page arrives complete, with its own address for each service, case and article. [Need this done? AI-Ready Website Rebuild](https://webrevolution.si/services/ai-ready-website/) **Contents 14 sections** My old agency website had services, testimonials and a project form. People saw a finished page. The programs behind ChatGPT, Claude and Perplexity received 0 words of it. That matters because buyers now ask an assistant first. 51% of B2B software buyers start their research with an AI chatbot more often than with a search engine, according to [G2’s buyer survey from March 2026](https://www.prnewswire.com/news-releases/new-g2-research-half-of-b2b-software-buyers-now-start-their-research-with-ai-chatbots-302742807.html). Call it AI or Super Intelligence (SI): the assistant can only describe what it can read. Below: what happened, whether it affects your site, a two-minute check, and what fixed mine. If you’d rather have it done for you, that’s my [AI-ready website rebuild for service firms](https://webrevolution.si/services/ai-ready-website/). ## The short version - Some websites are built inside the visitor’s browser. People see a full page; most AI assistants get an almost empty one. - Mine was one: AI crawlers got 0 words of my homepage. - Google usually reads these sites fine, so owners rarely notice. - You can check your own site in two minutes, with no technical skill. - The fix is a site where every page arrives complete. If yours already does, the cause is elsewhere. ## What happened to my own website? It looked finished to people, but the programs AI assistants use to read websites received an empty page. Until this site launched, my agency site was aiwebrevolution.com, built with Bolt.new, an AI tool that writes web apps. It was a single-page app: the server sends a nearly empty file plus a program, and the program builds the page in the visitor’s browser. People’s browsers run the program. Anything that doesn’t run it sees the empty file. On 5 October 2026 I fetched my homepage the way an AI crawler does, without a browser. Where the text should have been sat one empty container. 0 words of page content visible to non-JavaScript crawlers observed 2026-10-05 · raw HTML fetch 2 URLs in the sitemap observed 2026-10-05 · sitemap.xml What an AI assistant could learn from that: - **A page title and a one-line description.** Enough to know a business existed, not what it sold or to whom. - **A hidden description of a page that wasn’t there.** Structured data is labels in the code that tell machines who you are and what you sell. Mine sat above an empty page. - **Nothing to quote.** No services, prices, cases or articles. - **A sitemap with 2 entries.** That’s the list of pages a site hands to search engines; the app kept every other page inside the browser. - **No “page not found”.** Every mistyped address returned the same empty page. Worse, the hidden description was older than the site people saw: other years of experience, services and prices, plus a review rating I had published about myself. A machine got a confident, outdated story with nothing to check it against. ## Why can’t AI assistants read a site like that? Because the programs they send take the page as it arrives. They don’t run the code that builds it. Those programs are called crawlers: software that reads pages for a search engine or an AI assistant. In the largest public study of crawler logs, [published by Vercel and MERJ in December 2024](https://vercel.com/blog/the-rise-of-the-ai-crawler), none of the major AI crawlers ran that code. A smaller test by RESONEO in January 2026 found the same when a user pastes a link into ChatGPT or Claude. | Where your buyer asks | Can it read a page built in the browser? | | --- | --- | | ChatGPT (its search, its training, a pasted link) | No | | Claude (its training, a pasted link) | No. Its search crawler hasn’t been tested; I assume no | | Perplexity | No | | Google Search, AI Overviews and Gemini | Yes, through Google’s index. Gemini reading a pasted link: no | | Bing and Copilot | Generally yes | | Siri, Spotlight and Safari (Apple) | Yes | Sources: Vercel/MERJ, RESONEO, and Bing’s own blog. The pasted link matters most. When a buyer pastes your address into ChatGPT and asks what your firm does, ChatGPT fetches the page right then. On a site like my old one, it gets the empty page. ### Why do so many owners never notice? Google runs the code, so these sites look normal on Google, and owners assume every machine reads them that way. Even Google’s [guide for developers](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) recommends sending finished pages, because not every crawler runs the code. ## Does this affect my business? Only if your site builds its pages in the browser. Many don’t: a classic WordPress site, for example, sends its text with the page. The sites at risk are usually built as apps, by an agency using a modern app framework or by an AI app builder, as mine was. You don’t need to know which kind you have: the check below tells you in two minutes. ## What could it cost a firm like mine? Usually nothing you can see. The cost is enquiries that never start. Picture a 14-person accounting firm in Hamburg. A few years ago an agency rebuilt its website as a modern app. It looks sharp, and it shows up on Google for the firm’s name. A finance director at a manufacturer asks ChatGPT which accountants in Hamburg handle VAT for exporters. ChatGPT’s search draws on pages its crawler could read. The firm’s pages gave it nothing, so the answer names other firms. A week later a managing partner hears the firm’s name, pastes the address into ChatGPT and asks what the firm does. The answer is pieced together from a directory listing and an old review: vague, and partly wrong. Who notices? Nobody. There is no error message, Google still lists the firm, and the website statistics look normal. Say the firm wins about 20 new business clients a year, and two or three would have started with an assistant. At a few thousand euros in fees each per year, that is a real sum to lose quietly. In a case like this, the fix typically means the assistant can read the firm’s own description. Whether it then recommends the firm depends on its pages and reputation; nobody can promise that. ## How do I check if my website is invisible to ChatGPT? Three ways, quickest first. The first takes two minutes and needs no technical skill. **1. Look at the page as it arrives.** Open your homepage in Chrome or Edge, right-click an empty spot and choose “View page source”. Press Ctrl+F (Cmd+F on a Mac) and search for a sentence from the middle of the page. If it isn’t there, a crawler that doesn’t run code never sees it. Don’t use “Inspect”: it shows the page after the code has run. **2. Ask an assistant to read it.** Paste your address into ChatGPT and ask it to list the services on that page, in the page’s own words. An empty page, or an answer drawn only from other websites, is a warning sign. It’s a hint, not proof. **3. Run the free audit.** It fetches your homepage the way a crawler does and shows you what arrives, then checks your crawler settings, page list and structured data. *Form · GET /audit/ · fields: Your website address · button: “Scan my site →”* **Recognise this? See what AI sees on your site.** Free AI-readiness audit · about 60 seconds. ### How to read what you find - **Almost no words, but the page looks full in a browser:** this article’s problem. Your text needs to arrive with the page. - **Words there, but crawlers turned away:** an access problem. robots.txt (a small file that tells crawlers what they may read) or your hosting’s bot protection says no. That’s settings, not a rebuild. - **Words there, crawlers allowed, still not named in answers:** an authority problem. You need pages that answer buyers’ questions, and mentions on other sites. ## How did I fix my own site? I rebuilt it so that a crawler that doesn’t run code gets the same page a person sees. | | Before (aiwebrevolution.com) | After (webrevolution.si) | | --- | --- | --- | | Words an AI crawler reads | 0 | All the page text | | Pages with their own address | One real page; 2 in the sitemap | One per service, case, price list and article | | Description for machines | Outdated, with a rating I gave myself | Built from the same data as the visible text | | A mistyped address | The same empty page | A real “page not found” | | Next step for a visitor | One form inside the app | A plain project form, the free audit, booking on click | Two changes matter most for a business. **A page for everything I sell.** Each service, case and article has its own page, and each service shows its price. Assistants quote and link to specific pages. My old homepage had to answer every question, and as an empty shell it answered none. **One honest description for machines.** The structured data comes from the same file as the visible text, so the two can’t drift apart. Google’s [guide to its AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) says it isn’t required for them; I treat it as clarity, not a trick. ### What I rejected - **Showing crawlers a separate, pre-built copy of the old site.** Google calls this [a workaround, not a long-term solution](https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering), and it means two versions of every page to keep in step. - **Keeping the app and building each page on the server at every visit.** My site is documents plus a few forms; that would run code on every visit for text that changes weekly at most. ### What I can’t show you yet This site is as new as this article, so there are no results to show yet. I’m measuring which AI crawlers fetch which pages, what Google and Bing index, whether the main assistants name the site for a fixed set of buyer questions, and how many project letters they bring. I’ll publish the 90-day numbers here, whatever they show. ## Has this approach worked on other sites? Yes. The rebuild of this site is new, but the pattern isn’t. SitShape is my own sofa marketplace and shape configurator ([sitshape.com](https://sitshape.com/)). It now serves 564 ready-made pages in 3 languages. Before that rebuild, in a short live sample, all the server’s work came from one crawler walking product pages. Now those pages are plain files: 103 → 0 server runs, before and after. For a business, crawlers read the same finished pages people do, at no cost: Cloudflare doesn’t bill requests served from plain files. From the field OWN PRODUCT · LIVE ↗ SitShape · static marketplace My own sofa marketplace, rebuilt as ready-made pages: crawlers get the whole page at once and the server barely notices them. 564 ready-made pages in 3 languages observed 2026-10-01 · build output 103 → 0 server calls from crawlers in a 45-second sample, after the rebuild into ready-made pages observed 2026-10-01 · live sample [See how it was built](https://webrevolution.si/work/sitshape/) For Corvane Legal, a German law firm (name changed), I built an AI content engine that now has 1,100+ pages live on the firm's own domain. It recognises 16 crawlers by name, AI and search, and counts their visits daily, so the firm sees which bots read its pages instead of guessing. It has run in the firm’s own Cloudflare account since September 2026; the [law-firm content engine case study](https://webrevolution.si/work/law-firm-ai-content-engine/) has the details. ## Do I need special files for AI, like llms.txt? Not for Google, and probably only a little for anyone else. llms.txt is a proposal Jeremy Howard published in September 2024: a short file that summarises a site for AI models. Google says its Search doesn’t use such files; they neither help nor harm. My site has one anyway, because it costs nothing extra and some AI tools look for it. My free audit scores it as low impact. It won’t fix an empty site: the pages themselves have to carry the text. ## Which settings decide whether AI crawlers get in? If your site sits behind Cloudflare, three switches do. Being readable is one gate; being let in is the other. Cloudflare sorts AI traffic into Search (assistants looking things up), Agent (assistants acting for a person) and Training (collecting text to train AI models). Since 1 July 2026 every plan can allow or block each. Mine are all on Allow, because a sales site is written to be quoted; a publisher might choose otherwise. To refuse training but stay in Google’s and Bing’s search, a [newer Training option](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) does that. Details: [is Cloudflare blocking AI crawlers on your site](https://webrevolution.si/insights/cloudflare-blocking-ai-crawlers/). ## What you can do this week 1. **View your homepage’s source** and search for one sentence from the middle. About 2 minutes. 2. **Repeat it for a service page and an article**, since parts of a site can be built differently. About 5 minutes. 3. **Paste your address into ChatGPT**, ask it to list your services, and save the answer to compare after any fix. About 5 minutes. 4. **Run the free audit** above and read what a crawler sees. About 5 minutes. 5. **Ask your web developer:** does every page arrive with its text, without JavaScript? Send the section below with it. About 10 minutes. 6. **If you use Cloudflare,** ask whoever manages it how the three AI crawler settings are set. About 10 minutes. ## When is a rebuild worth it, and when isn’t it? It’s worth it when your pages look full but arrive empty, and you want your services, prices and cases in pages any machine can read. The rebuild is priced from €6,500 · from $7,200, typically €8k–16k · $9k–17.5k, takes 3–6 weeks, and is handed over into your own accounts. All prices are on [the pricing page](https://webrevolution.si/pricing/). It’s probably not for you if your page source already shows your text. Then the cause is likely access or authority, and a rebuild is the wrong spend. Start with the free audit, or the [AI Visibility Deep Audit](https://webrevolution.si/services/ai-visibility-audit/) at €1,500 · $1,650. Fixed price. What you pay in cash is credited in full to any build within 60 days. It’s also not for logged-in apps: nothing behind a login needs to be crawled. ## For your web developer - **The test.** Fetch without rendering and count visible body words (the title, JSON-LD and inline scripts survive tag-stripping); my old homepage returns 0. ```bash curl -sL -A "GPTBot" https://yourcompany.com/ \ | perl -0777 -pe 's/<(script|style|noscript|template)\b.*?<\/\1>//gis; s///gis; s/<[^>]+>/ /g' \ | wc -w ``` - **A 403 or challenge** only with `-A "GPTBot"`: the CDN reacts to the name, while real crawlers are verified by IP. An access question, not rendering. - **Who renders.** Not GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Meta-ExternalAgent or Bytespider (Vercel/MERJ; ChatGPT’s and Claude’s crawlers fetched JS files, 11.50% and 23.84% of requests, without running them), nor Claude-User (RESONEO). Claude-SearchBot is untested. Googlebot (via a queue), Bingbot (generally) and Applebot render; Gemini’s live fetch didn’t, Copilot’s did (RESONEO). - **The fix.** Static or server-side rendering, a URL per service, case and article, real 404s, JSON-LD from the same data as the copy. Google calls dynamic rendering a workaround. - **This site.** Astro builds every page to HTML, served from Workers static assets; the Worker passes HTML through (it only sets the currency) and runs forms, scans, payments and admin. Markdown twin per page (`index.md`), plus llms.txt. A JS budget fails the build; Core Web Vitals held to Google’s “good” thresholds, the H1 as LCP element. - **robots.txt.** One `User-agent: *` group with `Content-Signal: search=yes, ai-input=yes, ai-train=yes`, no named-bot groups (RFC 9309: a crawler that finds its own group follows only that one). Bot Preference Sync is off, so the file is served as written. ## What’s the next step? Start with the two-minute check. If your text arrives with the page, this problem isn’t yours, and the free audit shows where to look instead. If it arrives empty, that is fixable. Found by AI · done for you Your site looks finished in a browser, but AI crawlers receive an empty shell. I rebuild it as static HTML on Cloudflare, with a real page for every service, case and article, crawler settings checked, and everything handed over into your own accounts. from €6,500 · $7,200 3–6 weeks [Start with this →](https://webrevolution.si/start/?goal=rebuild&ref=website-invisible-to-chatgpt) [See what's included](https://webrevolution.si/services/ai-ready-website/) For everything “AI-ready” covers beyond this, read [what an AI-ready website is in 2026](https://webrevolution.si/insights/what-is-an-ai-ready-website/). Or [write me a letter](https://webrevolution.si/start/) with your address. I look at your site before I reply, so my first answer is already useful. ## Sources Every factual claim above traces to one of these. Numbers about my own work come from my dated receipts. 1. [Vercel and MERJ: The rise of the AI crawler (17 December 2024)](https://vercel.com/blog/the-rise-of-the-ai-crawler) vercel.com 2. [RESONEO: AI search crawler test results, LLM rendering capabilities (January 2026)](https://think.resoneo.com/sentinel/geo-llm-crawler-report.html) think.resoneo.com 3. [OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)](https://developers.openai.com/docs/gptbot) developers.openai.com 4. [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) support.claude.com 5. [Perplexity: Perplexity Crawlers](https://docs.perplexity.ai/guides/bots) docs.perplexity.ai 6. [Apple: About Applebot](https://support.apple.com/en-us/119829) support.apple.com 7. [Bing Webmaster Blog: bingbot Series: JavaScript, Dynamic Rendering, and Cloaking. Oh My! (31 October 2018)](https://blogs.bing.com/webmaster/october-2018/bingbot-Series-JavaScript,-Dynamic-Rendering,-and-Cloaking-Oh-My) blogs.bing.com 8. [Google Search Central: Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) developers.google.com 9. [Google Search Central: Dynamic rendering as a workaround](https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering) developers.google.com 10. [Google Search Central: Optimizing your website for generative AI features on Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) developers.google.com 11. [Cloudflare changelog: New options to manage AI traffic (1 July 2026)](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/) developers.cloudflare.com 12. [Cloudflare blog: Say it once: introducing Bot Preference Sync (21 August 2026)](https://blog.cloudflare.com/bot-preference-sync/) blog.cloudflare.com 13. [Cloudflare blog: Have it both ways: stay discoverable in search while disallowing AI training (15 September 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) blog.cloudflare.com 14. [Cloudflare blog: Giving users choice with Cloudflare's new Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) blog.cloudflare.com 15. [IETF RFC 9309: Robots Exclusion Protocol, section 2.2.1 (the user-agent line)](https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1) rfc-editor.org 16. [Cloudflare docs: Workers static assets](https://developers.cloudflare.com/workers/static-assets/) developers.cloudflare.com 17. [Cloudflare docs: Workers static assets, billing and limitations](https://developers.cloudflare.com/workers/static-assets/billing-and-limitations/) developers.cloudflare.com 18. [llms.txt: the proposal (Jeremy Howard, September 2024)](https://llmstxt.org/) llmstxt.org 19. [web.dev: Web Vitals](https://web.dev/articles/vitals) web.dev 20. [G2 via PR Newswire: Half of B2B software buyers now start their research with AI chatbots (April 2026; survey fielded March 2026)](https://www.prnewswire.com/news-releases/new-g2-research-half-of-b2b-software-buyers-now-start-their-research-with-ai-chatbots-302742807.html) prnewswire.com ## Changelog - 8 Nov 2026 First published. Reviewed by Vegard Wikeby on 8 Nov 2026. The Updated date changes only when the substance changes, never for a typo. ![Vegard Wikeby](https://webrevolution.si/_astro/vegard-headshot.CXuTy1_D_ZsxVMd.webp) About the author [Vegard Wikeby](https://webrevolution.si/about/) I shipped my first e-commerce shop in 2006 and have worked independently since 2014. Today I direct a crew of AI agents that research, draft and test, and I decide what ships. This article was made the same way: agents gathered the sources, I checked every claim against them. [Book 20 minutes](https://webrevolution.si/contact/#book) [Write to me](https://webrevolution.si/start/?ref=website-invisible-to-chatgpt) Personal reply within one business day. Agents propose. A human approves. [**Vegard Wikeby** One man. An army of agents. I write these myself, with research agents.](https://webrevolution.si/about/) Related service [AI-Ready Website Rebuild](https://webrevolution.si/services/ai-ready-website/) Your site rebuilt so AI crawlers read every word, agents can act on it and the people they send convert. from €6,500 · $7,200 3–6 weeks [Start with this](https://webrevolution.si/start/?goal=rebuild&ref=website-invisible-to-chatgpt) See what AI sees on your site Free AI-readiness audit. About 60 seconds. *Form · GET /audit/ · fields: Your website address · button: “Scan my site →”* Keep reading ## More on AI-ready websites [AI-ready websites](https://webrevolution.si/insights/topic/ai-ready-websites/) 8 Nov 2026 · 10 min read · 16 sources ### [Is Cloudflare Blocking AI Crawlers on Your Website? How to Check, and What to Ask Your Developer](https://webrevolution.si/insights/cloudflare-blocking-ai-crawlers/) A security setting at Cloudflare or your web host can quietly turn ChatGPT, Claude and Perplexity away. How to check in minutes, and what to ask your developer. [AI-ready websites](https://webrevolution.si/insights/topic/ai-ready-websites/) 8 Nov 2026 · 11 min read · 23 sources ### [What Is an AI-Ready Website? What It Means for Your Business and What to Fix First](https://webrevolution.si/insights/what-is-an-ai-ready-website/) What an AI-ready website means for a B2B firm, how to tell in an afternoon whether AI assistants can find and quote yours, and what to fix first. Super Intelligence (SI) for business · monthly ## One email a month. What changed in AI, and what to do about it. The month’s new articles plus a short, sourced digest of what moved in AI for B2B businesses. Researched by agents, approved line by line by me before it is sent. [See a sample issue](https://webrevolution.si/briefing/) *Form · POST /api/briefing/subscribe · fields: Your work email · button: “Get the SI Briefing”* One email a month. Unsubscribe in one click. You confirm by email first. --- ## Structured facts The page's JSON-LD (schema.org), flattened to text. ### WebSite: WebRevolution.si - url: https://webrevolution.si/ - inLanguage: en - publisher: WebRevolution.si ### OnlineBusiness: WebRevolution.si - alternateName: WebRevolution; webrevolution.si; WebRevolution SI - url: https://webrevolution.si/ - logo: https://webrevolution.si/icon-512.png - slogan: One man. An army of agents. - description: One-man AI agency for B2B: AI-ready websites, AI content & distribution engines, AI agents and custom tools that AI assistants can find, cite and recommend. - founder: Vegard Wikeby - areaServed: Worldwide - knowsAbout: Generative engine optimization; AI-ready websites; AI agents for business; AI content engines; Cloudflare Workers; Conversion rate optimization; Model Context Protocol - contactPoint: ContactPoint (contactType: sales; url: https://webrevolution.si/contact/; availableLanguage: en) - sameAs: https://www.linkedin.com/in/vegardwikeby; https://x.com/vegardwikeby ### Person: Vegard Wikeby - url: https://webrevolution.si/about/ - jobTitle: Founder, WebRevolution.si - worksFor: WebRevolution.si - image: https://webrevolution.si/img/vegard-wikeby.jpg - sameAs: https://www.linkedin.com/in/vegardwikeby; https://x.com/vegardwikeby - knowsAbout: Web development; AI agents; Generative engine optimization; WordPress; E-commerce; Cloudflare ### BreadcrumbList - Home › Insights › AI-ready websites › My Website Was Invisible to ChatGPT ### BlogPosting: My Website Was Invisible to ChatGPT: It Read 0 Words. How to Check Yours, and What I Changed - description: My old website showed AI assistants 0 words. What that means for your firm, how to check your own site in two minutes, and what fixed mine. - url: https://webrevolution.si/insights/website-invisible-to-chatgpt/ - mainEntityOfPage: https://webrevolution.si/insights/website-invisible-to-chatgpt/ - image: https://webrevolution.si/og/insights/website-invisible-to-chatgpt.png - inLanguage: en - keywords: website invisible to ChatGPT - articleSection: AI-ready websites - wordCount: 2548 - datePublished: 2026-11-08 - dateModified: 2026-11-08 - author: Vegard Wikeby - publisher: WebRevolution.si - isPartOf: CollectionPage (name: AI-ready websites: readable by crawlers, usable by agents; url: https://webrevolution.si/insights/topic/ai-ready-websites/) - citation: - CreativeWork (name: Vercel and MERJ: The rise of the AI crawler (17 December 2024); url: https://vercel.com/blog/the-rise-of-the-ai-crawler) - CreativeWork (name: RESONEO: AI search crawler test results, LLM rendering capabilities (January 2026); url: https://think.resoneo.com/sentinel/geo-llm-crawler-report.html) - CreativeWork (name: OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User); url: https://developers.openai.com/docs/gptbot) - CreativeWork (name: Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?; url: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) - CreativeWork (name: Perplexity: Perplexity Crawlers; url: https://docs.perplexity.ai/guides/bots) - CreativeWork (name: Apple: About Applebot; url: https://support.apple.com/en-us/119829) - CreativeWork (name: Bing Webmaster Blog: bingbot Series: JavaScript, Dynamic Rendering, and Cloaking. Oh My! (31 October 2018); url: https://blogs.bing.com/webmaster/october-2018/bingbot-Series-JavaScript,-Dynamic-Rendering,-and-Cloaking-Oh-My) - CreativeWork (name: Google Search Central: Understand JavaScript SEO basics; url: https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) - CreativeWork (name: Google Search Central: Dynamic rendering as a workaround; url: https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering) - CreativeWork (name: Google Search Central: Optimizing your website for generative AI features on Google Search; url: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) - CreativeWork (name: Cloudflare changelog: New options to manage AI traffic (1 July 2026); url: https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/) - CreativeWork (name: Cloudflare blog: Say it once: introducing Bot Preference Sync (21 August 2026); url: https://blog.cloudflare.com/bot-preference-sync/) - CreativeWork (name: Cloudflare blog: Have it both ways: stay discoverable in search while disallowing AI training (15 September 2026); url: https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) - CreativeWork (name: Cloudflare blog: Giving users choice with Cloudflare's new Content Signals Policy; url: https://blog.cloudflare.com/content-signals-policy/) - CreativeWork (name: IETF RFC 9309: Robots Exclusion Protocol, section 2.2.1 (the user-agent line); url: https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1) - CreativeWork (name: Cloudflare docs: Workers static assets; url: https://developers.cloudflare.com/workers/static-assets/) - CreativeWork (name: Cloudflare docs: Workers static assets, billing and limitations; url: https://developers.cloudflare.com/workers/static-assets/billing-and-limitations/) - CreativeWork (name: llms.txt: the proposal (Jeremy Howard, September 2024); url: https://llmstxt.org/) - CreativeWork (name: web.dev: Web Vitals; url: https://web.dev/articles/vitals) - CreativeWork (name: G2 via PR Newswire: Half of B2B software buyers now start their research with AI chatbots (April 2026; survey fielded March 2026); url: https://www.prnewswire.com/news-releases/new-g2-research-half-of-b2b-software-buyers-now-start-their-research-with-ai-chatbots-302742807.html)