Contents14 sections
My old agency website had services, testimonials and a project form. People saw a finished page. The programs behind ChatGPT, Claude and Perplexity received 0 words of it.
That matters because buyers now ask an assistant first. 51% of B2B software buyers start their research with an AI chatbot more often than with a search engine, according to G2’s buyer survey from March 2026. Call it AI or Super Intelligence (SI): the assistant can only describe what it can read.
Below: what happened, whether it affects your site, a two-minute check, and what fixed mine. If you’d rather have it done for you, that’s my AI-ready website rebuild for service firms.
The short version
- Some websites are built inside the visitor’s browser. People see a full page; most AI assistants get an almost empty one.
- Mine was one: AI crawlers got 0 words of my homepage.
- Google usually reads these sites fine, so owners rarely notice.
- You can check your own site in two minutes, with no technical skill.
- The fix is a site where every page arrives complete. If yours already does, the cause is elsewhere.
What happened to my own website?
It looked finished to people, but the programs AI assistants use to read websites received an empty page.
Until this site launched, my agency site was aiwebrevolution.com, built with Bolt.new, an AI tool that writes web apps. It was a single-page app: the server sends a nearly empty file plus a program, and the program builds the page in the visitor’s browser. People’s browsers run the program. Anything that doesn’t run it sees the empty file.
On 5 October 2026 I fetched my homepage the way an AI crawler does, without a browser. Where the text should have been sat one empty container.
What an AI assistant could learn from that:
- A page title and a one-line description. Enough to know a business existed, not what it sold or to whom.
- A hidden description of a page that wasn’t there. Structured data is labels in the code that tell machines who you are and what you sell. Mine sat above an empty page.
- Nothing to quote. No services, prices, cases or articles.
- A sitemap with 2 entries. That’s the list of pages a site hands to search engines; the app kept every other page inside the browser.
- No “page not found”. Every mistyped address returned the same empty page.
Worse, the hidden description was older than the site people saw: other years of experience, services and prices, plus a review rating I had published about myself. A machine got a confident, outdated story with nothing to check it against.
Why can’t AI assistants read a site like that?
Because the programs they send take the page as it arrives. They don’t run the code that builds it.
Those programs are called crawlers: software that reads pages for a search engine or an AI assistant. In the largest public study of crawler logs, published by Vercel and MERJ in December 2024, none of the major AI crawlers ran that code. A smaller test by RESONEO in January 2026 found the same when a user pastes a link into ChatGPT or Claude.
| Where your buyer asks | Can it read a page built in the browser? |
|---|---|
| ChatGPT (its search, its training, a pasted link) | No |
| Claude (its training, a pasted link) | No. Its search crawler hasn’t been tested; I assume no |
| Perplexity | No |
| Google Search, AI Overviews and Gemini | Yes, through Google’s index. Gemini reading a pasted link: no |
| Bing and Copilot | Generally yes |
| Siri, Spotlight and Safari (Apple) | Yes |
Sources: Vercel/MERJ, RESONEO, and Bing’s own blog.
The pasted link matters most. When a buyer pastes your address into ChatGPT and asks what your firm does, ChatGPT fetches the page right then. On a site like my old one, it gets the empty page.
Why do so many owners never notice?
Google runs the code, so these sites look normal on Google, and owners assume every machine reads them that way. Even Google’s guide for developers recommends sending finished pages, because not every crawler runs the code.
Does this affect my business?
Only if your site builds its pages in the browser. Many don’t: a classic WordPress site, for example, sends its text with the page.
The sites at risk are usually built as apps, by an agency using a modern app framework or by an AI app builder, as mine was. You don’t need to know which kind you have: the check below tells you in two minutes.
What could it cost a firm like mine?
Usually nothing you can see. The cost is enquiries that never start.
Picture a 14-person accounting firm in Hamburg. A few years ago an agency rebuilt its website as a modern app. It looks sharp, and it shows up on Google for the firm’s name.
A finance director at a manufacturer asks ChatGPT which accountants in Hamburg handle VAT for exporters. ChatGPT’s search draws on pages its crawler could read. The firm’s pages gave it nothing, so the answer names other firms.
A week later a managing partner hears the firm’s name, pastes the address into ChatGPT and asks what the firm does. The answer is pieced together from a directory listing and an old review: vague, and partly wrong.
Who notices? Nobody. There is no error message, Google still lists the firm, and the website statistics look normal. Say the firm wins about 20 new business clients a year, and two or three would have started with an assistant. At a few thousand euros in fees each per year, that is a real sum to lose quietly.
In a case like this, the fix typically means the assistant can read the firm’s own description. Whether it then recommends the firm depends on its pages and reputation; nobody can promise that.
How do I check if my website is invisible to ChatGPT?
Three ways, quickest first. The first takes two minutes and needs no technical skill.
1. Look at the page as it arrives. Open your homepage in Chrome or Edge, right-click an empty spot and choose “View page source”. Press Ctrl+F (Cmd+F on a Mac) and search for a sentence from the middle of the page. If it isn’t there, a crawler that doesn’t run code never sees it. Don’t use “Inspect”: it shows the page after the code has run.
2. Ask an assistant to read it. Paste your address into ChatGPT and ask it to list the services on that page, in the page’s own words. An empty page, or an answer drawn only from other websites, is a warning sign. It’s a hint, not proof.
3. Run the free audit. It fetches your homepage the way a crawler does and shows you what arrives, then checks your crawler settings, page list and structured data.
How to read what you find
- Almost no words, but the page looks full in a browser: this article’s problem. Your text needs to arrive with the page.
- Words there, but crawlers turned away: an access problem. robots.txt (a small file that tells crawlers what they may read) or your hosting’s bot protection says no. That’s settings, not a rebuild.
- Words there, crawlers allowed, still not named in answers: an authority problem. You need pages that answer buyers’ questions, and mentions on other sites.
How did I fix my own site?
I rebuilt it so that a crawler that doesn’t run code gets the same page a person sees.
| Before (aiwebrevolution.com) | After (webrevolution.si) | |
|---|---|---|
| Words an AI crawler reads | 0 | All the page text |
| Pages with their own address | One real page; 2 in the sitemap | One per service, case, price list and article |
| Description for machines | Outdated, with a rating I gave myself | Built from the same data as the visible text |
| A mistyped address | The same empty page | A real “page not found” |
| Next step for a visitor | One form inside the app | A plain project form, the free audit, booking on click |
Two changes matter most for a business.
A page for everything I sell. Each service, case and article has its own page, and each service shows its price. Assistants quote and link to specific pages. My old homepage had to answer every question, and as an empty shell it answered none.
One honest description for machines. The structured data comes from the same file as the visible text, so the two can’t drift apart. Google’s guide to its AI features says it isn’t required for them; I treat it as clarity, not a trick.
What I rejected
- Showing crawlers a separate, pre-built copy of the old site. Google calls this a workaround, not a long-term solution, and it means two versions of every page to keep in step.
- Keeping the app and building each page on the server at every visit. My site is documents plus a few forms; that would run code on every visit for text that changes weekly at most.
What I can’t show you yet
This site is as new as this article, so there are no results to show yet. I’m measuring which AI crawlers fetch which pages, what Google and Bing index, whether the main assistants name the site for a fixed set of buyer questions, and how many project letters they bring. I’ll publish the 90-day numbers here, whatever they show.
Has this approach worked on other sites?
Yes. The rebuild of this site is new, but the pattern isn’t.
SitShape is my own sofa marketplace and shape configurator (sitshape.com). It now serves 564 ready-made pages in 3 languages. Before that rebuild, in a short live sample, all the server’s work came from one crawler walking product pages. Now those pages are plain files: 103 → 0 server runs, before and after. For a business, crawlers read the same finished pages people do, at no cost: Cloudflare doesn’t bill requests served from plain files.
For Corvane Legal, a German law firm (name changed), I built an AI content engine that now has 1,100+ pages live on the firm's own domain. It recognises 16 crawlers by name, AI and search, and counts their visits daily, so the firm sees which bots read its pages instead of guessing. It has run in the firm’s own Cloudflare account since September 2026; the law-firm content engine case study has the details.
Do I need special files for AI, like llms.txt?
Not for Google, and probably only a little for anyone else.
llms.txt is a proposal Jeremy Howard published in September 2024: a short file that summarises a site for AI models. Google says its Search doesn’t use such files; they neither help nor harm. My site has one anyway, because it costs nothing extra and some AI tools look for it. My free audit scores it as low impact. It won’t fix an empty site: the pages themselves have to carry the text.
Which settings decide whether AI crawlers get in?
If your site sits behind Cloudflare, three switches do. Being readable is one gate; being let in is the other.
Cloudflare sorts AI traffic into Search (assistants looking things up), Agent (assistants acting for a person) and Training (collecting text to train AI models). Since 1 July 2026 every plan can allow or block each. Mine are all on Allow, because a sales site is written to be quoted; a publisher might choose otherwise. To refuse training but stay in Google’s and Bing’s search, a newer Training option does that. Details: is Cloudflare blocking AI crawlers on your site.
What you can do this week
- View your homepage’s source and search for one sentence from the middle. About 2 minutes.
- Repeat it for a service page and an article, since parts of a site can be built differently. About 5 minutes.
- Paste your address into ChatGPT, ask it to list your services, and save the answer to compare after any fix. About 5 minutes.
- Run the free audit above and read what a crawler sees. About 5 minutes.
- Ask your web developer: does every page arrive with its text, without JavaScript? Send the section below with it. About 10 minutes.
- If you use Cloudflare, ask whoever manages it how the three AI crawler settings are set. About 10 minutes.
When is a rebuild worth it, and when isn’t it?
It’s worth it when your pages look full but arrive empty, and you want your services, prices and cases in pages any machine can read.
The rebuild is priced from €6,500from $7,200, typically €8k–16k$9k–17.5k, takes 3–6 weeks, and is handed over into your own accounts. All prices are on the pricing page.
It’s probably not for you if your page source already shows your text. Then the cause is likely access or authority, and a rebuild is the wrong spend. Start with the free audit, or the AI Visibility Deep Audit at €1,500$1,650. Fixed price. What you pay in cash is credited in full to any build within 60 days. It’s also not for logged-in apps: nothing behind a login needs to be crawled.
For your web developer
- The test. Fetch without rendering and count visible body words (the title, JSON-LD and inline scripts survive tag-stripping); my old homepage returns 0.
curl -sL -A "GPTBot" https://yourcompany.com/ \
| perl -0777 -pe 's/<(script|style|noscript|template)\b.*?<\/\1>//gis; s/<head\b.*?<\/head>//gis; s/<[^>]+>/ /g' \
| wc -w
- A 403 or challenge only with
-A "GPTBot": the CDN reacts to the name, while real crawlers are verified by IP. An access question, not rendering. - Who renders. Not GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Meta-ExternalAgent or Bytespider (Vercel/MERJ; ChatGPT’s and Claude’s crawlers fetched JS files, 11.50% and 23.84% of requests, without running them), nor Claude-User (RESONEO). Claude-SearchBot is untested. Googlebot (via a queue), Bingbot (generally) and Applebot render; Gemini’s live fetch didn’t, Copilot’s did (RESONEO).
- The fix. Static or server-side rendering, a URL per service, case and article, real 404s, JSON-LD from the same data as the copy. Google calls dynamic rendering a workaround.
- This site. Astro builds every page to HTML, served from Workers static assets; the Worker passes HTML through (it only sets the currency) and runs forms, scans, payments and admin. Markdown twin per page (
index.md), plus llms.txt. A JS budget fails the build; Core Web Vitals held to Google’s “good” thresholds, the H1 as LCP element. - robots.txt. One
User-agent: *group withContent-Signal: search=yes, ai-input=yes, ai-train=yes, no named-bot groups (RFC 9309: a crawler that finds its own group follows only that one). Bot Preference Sync is off, so the file is served as written.
What’s the next step?
Start with the two-minute check. If your text arrives with the page, this problem isn’t yours, and the free audit shows where to look instead. If it arrives empty, that is fixable.
For everything “AI-ready” covers beyond this, read what an AI-ready website is in 2026. Or write me a letter with your address. I look at your site before I reply, so my first answer is already useful.
Sources
Every factual claim above traces to one of these. Numbers about my own work come from my dated receipts.
- Vercel and MERJ: The rise of the AI crawler (17 December 2024) vercel.com
- RESONEO: AI search crawler test results, LLM rendering capabilities (January 2026) think.resoneo.com
- OpenAI: Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User) developers.openai.com
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler? support.claude.com
- Perplexity: Perplexity Crawlers docs.perplexity.ai
- Apple: About Applebot support.apple.com
- Bing Webmaster Blog: bingbot Series: JavaScript, Dynamic Rendering, and Cloaking. Oh My! (31 October 2018) blogs.bing.com
- Google Search Central: Understand JavaScript SEO basics developers.google.com
- Google Search Central: Dynamic rendering as a workaround developers.google.com
- Google Search Central: Optimizing your website for generative AI features on Google Search developers.google.com
- Cloudflare changelog: New options to manage AI traffic (1 July 2026) developers.cloudflare.com
- Cloudflare blog: Say it once: introducing Bot Preference Sync (21 August 2026) blog.cloudflare.com
- Cloudflare blog: Have it both ways: stay discoverable in search while disallowing AI training (15 September 2026) blog.cloudflare.com
- Cloudflare blog: Giving users choice with Cloudflare's new Content Signals Policy blog.cloudflare.com
- IETF RFC 9309: Robots Exclusion Protocol, section 2.2.1 (the user-agent line) rfc-editor.org
- Cloudflare docs: Workers static assets developers.cloudflare.com
- Cloudflare docs: Workers static assets, billing and limitations developers.cloudflare.com
- llms.txt: the proposal (Jeremy Howard, September 2024) llmstxt.org
- web.dev: Web Vitals web.dev
- G2 via PR Newswire: Half of B2B software buyers now start their research with AI chatbots (April 2026; survey fielded March 2026) prnewswire.com
Changelog
- First published.
Reviewed by Vegard Wikeby on . The Updated date changes only when the substance changes, never for a typo.