Can major AI crawlers actually reach a small business website?
A first-hand test on one live site, after a CDN was found silently disallowing eight training crawlers that nobody had chosen to disallow.
Status
Collection in progress. Partial data only. One site, measured repeatedly. The first full round is reported below. This is a case report, not a population study, and the sample size is one.
Why we ran it
While auditing our own site we found that its robots.txt was not the file in the repository. A CDN-level feature was injecting an additional managed block that disallowed GPTBot, ClaudeBot, CCBot, Google-Extended, Amazonbot, Applebot-Extended, Bytespider and meta-externalagent, directly above our own block that explicitly allowed the same agents.
Nobody had edited robots.txt. A platform default had been applied at the edge, and nothing in the deployment pipeline surfaced it. The file in version control and the file being served to crawlers were different files, and only one of them mattered.
We initially described this as a block on AI crawlers, and that description was wrong in an important way. Correcting it is the most useful thing on this page, so it has its own section below.
The question the incident raised is one we could not find measured anywhere: if you test it rather than assume it, can these crawlers reach a normal small business site, and would the owner have any way of knowing they could not?
Method
Nine crawler user agents were requested against eleven URLs each, for 99 crawler requests, plus eleven requests from an ordinary desktop browser user agent as a control, for 110 total. The agents were Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended.
The eleven URLs spanned page types rather than clustering on one: the homepage, an about page, a commercial hub, a service page, a deep industry page, a local page, an answers page, a contact page, and the three machine-readable files, llms.txt, robots.txt and sitemap.xml.
For every request we recorded the HTTP status, the response body length, and the number of JSON-LD blocks present. Body length was compared byte for byte against the browser control for the same URL, because the failure mode that matters most is not a block. It is being served a thinner page than a human sees, which no status code reveals.
Rate limiting was tested separately: 25 requests issued concurrently under the GPTBot user agent.
Results, first round
Measured 2026-09-03 at 18:36 UTC against the live production site.
All 99 crawler requests returned HTTP 200. Every response was byte-identical in length to the browser control for the same URL, and every one carried the same four JSON-LD blocks the browser received. No challenge pages, no soft blocks, no truncated bodies, and no differential serving of any kind.
The rate limit test returned 25 HTTP 200 responses in 1.1 seconds with no 429 and no 503.
Before the CDN block was removed, the same site was serving a robots.txt of 3,079 bytes containing a managed block and a Content-Signal directive. Afterwards it was 1,243 bytes and matched the repository exactly. The repository content had not changed.
| User agent | HTTP 200 | JSON-LD blocks | Body vs browser control |
|---|---|---|---|
| Googlebot | 11 / 11 | 4 | identical |
| Bingbot | 11 / 11 | 4 | identical |
| GPTBot | 11 / 11 | 4 | identical |
| OAI-SearchBot | 11 / 11 | 4 | identical |
| ChatGPT-User | 11 / 11 | 4 | identical |
| ClaudeBot | 11 / 11 | 4 | identical |
| Claude-SearchBot | 11 / 11 | 4 | identical |
| PerplexityBot | 11 / 11 | 4 | identical |
| Google-Extended | 11 / 11 | 4 | identical |
| Browser (control) | 11 / 11 | 4 | baseline |
What this does not show, which is most of what people want it to show
Access is the first link in a chain of five, and it is the only one this test measures. A crawler being able to fetch a page does not mean the page was fetched. Being fetched does not mean it was indexed. Being indexed does not mean it was retrieved for any query. Being retrieved does not mean it was cited. And being cited does not mean the business was recommended.
Every one of those transitions can fail independently, and this study speaks only to the first. We have seen the distinction collapsed repeatedly into claims that a site is "visible to ChatGPT" because a crawler received a 200. It is not the same statement.
It also cannot tell you these crawlers visited. A synthetic request with a spoofed user agent proves the server will answer such a request. Confirming real visits requires server logs and reverse DNS verification against the published IP ranges, which is a different test.
And the sample is one site, on one CDN, on one day. It shows the block existed and that removing it restored access. It does not establish how common the block is, which is the more interesting question and the one we cannot answer from here.
What the block actually did, which is not what we first said
Our first account of this incident said the site was blocking AI crawlers and implied the effect on AI search visibility was severe. Having since read the documentation properly, that was an overstatement, and the accurate version is more interesting.
Every user agent in that managed block is a TRAINING crawler. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the one that controls appearance in ChatGPT search, and states that the settings are independent. Anthropic documents three separate bots: ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for user-triggered retrieval. Google-Extended governs Gemini Apps and Vertex AI, not Google Search. Amazonbot, Applebot-Extended, Bytespider, CCBot and meta-externalagent are likewise not search crawlers.
The managed block contained no search crawler at all. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot were never disallowed. The block even carried an explicit signal setting search to yes and only AI training to no.
So what we actually had was a training opt-out applied by default, not a search block. Whether you want that opt-out is a real decision with reasonable arguments on both sides. What it was not is an outage in AI search visibility, and describing it as one would have been the convenient story rather than the true one.
The genuine finding survives intact and is narrower: the file in version control was not the file being served, the difference was invisible from inside the repository, and the only way to find it was to request the site as a crawler and read what came back. That is worth knowing regardless of which crawlers were in the list.
What this study cannot tell you
Published alongside the method, not buried after the results, because a study that hides its limits is advertising.
Sample size is one site. This is a case report and the language throughout reflects that.
User agents were spoofed by us. This measures how the server responds to those strings, not that the real crawlers visited. Verified visits require server logs and reverse DNS checks against published IP ranges.
One CDN, one configuration, one day. The frequency of this misconfiguration across the web is unknown and is not estimated here.
Access only. The study does not measure indexing, retrieval, citation or recommendation, and no claim about those is made from this data.
The pre-fix state was recorded as file size and content rather than as a full 99-request run, because the block was removed before this protocol was written. The before and after are therefore not perfectly symmetric.
Our own first interpretation of the incident was wrong and is corrected in the body above. It is left visible rather than quietly edited out, because a study that hides its own corrections is asking for a trust it has not earned.