The AI readiness audit bot: what it reads, and how to opt out

How it identifies itself

WebRevolutionAudit/1.0 (+https://webrevolution.si/audit/bot/)

It runs on Cloudflare’s network, so requests come from Cloudflare IP addresses, not a fixed range. It only runs when a person enters your address at webrevolution.si/audit/. There is no crawl schedule and no index.

What one audit fetches

• /robots.txtfirst, so a site that opts out is never fetched further
• The page you enteredone GET, the way an AI crawler reads it: raw HTML, no JavaScript
• /sitemap.xmlor the sitemap named in robots.txt, plus one child sitemap if it is an index
• /llms.txtone GET
• Up to 5 pages from your sitemapone GET each, to check authors, dates, prices and answers
• One made-up URLto see whether missing pages return a real 404
• /.well-known/mcp.json and /.well-known/agent-card.jsonone GET each, to look for agent endpoints
• Up to 3 of your own JavaScript filesone GET each, only to measure their size

Every response is capped at 2 MB and 10 seconds, the whole audit at 30 seconds. Redirects are followed for at most 3 hops. Results are cached for 24 hours, so a repeat scan within a day reuses them. The report link stays live for 12 months. A site is normally fetched once a day at most, however many people check it, and each connection is limited to 3 audits an hour.

The crawler user-agent probe

To show how your server treats AI crawlers, the audit sends one HEAD request to the page you entered with each of these user-agent strings, 8 requests in total. If your server refuses HEAD, it sends one small GET instead (read up to 16 KB).

• OAI-SearchBot
• ChatGPT-User
• GPTBot
• ClaudeBot
• Claude-SearchBot
• PerplexityBot
• Bingbot
• CCBot

These requests come from my audit server, not from OpenAI, Anthropic, Perplexity, Microsoft or Common Crawl, and the report says so. Real crawlers come from verified IP ranges that your CDN may treat differently. The probe shows what a firewall rule based on user-agents alone would do. Google-Extended and Applebot-Extended are robots.txt tokens only, so they are never impersonated.

How to opt out

Add this to your robots.txt. The audit reads robots.txt before anything else and stops when it finds it, telling the person who asked that your site opted out.

User-agent: WebRevolutionAudit
Disallow: /

A general “User-agent: *” block does not stop the audit, because each audit is requested by a person for one site, like a browser visit. Only a rule naming WebRevolutionAudit does.

Something wrong?

If the bot misbehaves on your site, tell me and I will look at it the same day. WhatsApp me or write to me.

WebRevolution.si · Vegard Wikeby