How do you test whether AI crawlers can access your website?
A dozen curl commands, one browser baseline, and an honest account of what this method cannot prove.
How do you test whether AI crawlers can access your website?
Request your own URLs with each crawler's published user-agent string and compare the result against an ordinary browser request. Compare response body length byte for byte, not just the status code, because a server can return 200 and still hand a crawler a thinner page than a person receives. curl does this in a single line. Be clear about the limit: spoofing a user agent proves how your own server responds to that string, and nothing more. Confirming that a real crawler visited requires server logs, reverse DNS, and the IP ranges each operator publishes.
What you are actually testing
There are two separate questions and they need two different methods. The first is whether your server would serve a crawler if one arrived. The second is whether crawlers are arriving.
Spoofing a user agent answers the first. It is a synthetic request that you made, and it tells you about your own stack: your CDN rules, your bot protection, your edge configuration, your robots.txt as served rather than as committed.
The second question can only be answered from your logs. No amount of curl proves that OpenAI or Anthropic fetched anything.
Both are worth doing. Conflating them is the mistake, and it is a common one.
Why status codes are the wrong pass condition
A 200 feels like a pass and often is not. The failure mode that costs the most is a page that returns successfully but with less content than a browser gets, because a rendering path, a paywall rule or a bot-mitigation service treated the request differently.
So the comparison is byte length against a browser control for the same URL. Identical length is a strong signal that the same document was served. A shorter body with a 200 status is the case you are hunting for, and it is invisible if you only log status codes.
Also count your structured data blocks in both responses. If the JSON-LD is present for a browser and missing for a crawler, that is a real defect, and it is one that no status code reveals.
| Agent | Published user-agent string or note | Verification endpoint |
|---|---|---|
| GPTBot | Short AppleWebKit string ending in compatible; GPTBot/1.4; +https://openai.com/gptbot | https://openai.com/gptbot.json |
| OAI-SearchBot | Full Chrome-style string ending in compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot | https://openai.com/searchbot.json |
| ChatGPT-User | Short AppleWebKit string ending in compatible; ChatGPT-User/1.0; +https://openai.com/bot | https://openai.com/chatgpt-user.json |
| ClaudeBot | Anthropic publishes no user-agent string, only the robots.txt token | https://claude.com/crawling/bots.json |
| Claude-SearchBot | Anthropic publishes no user-agent string, only the robots.txt token | https://claude.com/crawling/bots.json |
| PerplexityBot | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) | https://www.perplexity.ai/perplexitybot.json |
| Googlebot | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36 | reverse DNS to googlebot.com, plus common-crawlers.json |
| bingbot | Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) W.X.Y.Z Safari/537.36 | reverse then forward DNS to search.msn.com |
| CCBot | CCBot/2.0 (https://commoncrawl.org/faq/) | https://index.commoncrawl.org/ccbot.json |
| Applebot | Safari-style string ending in (Applebot/0.1; +http://www.apple.com/go/applebot) | https://search.developer.apple.com/applebot.json |
Read the robots.txt you serve, not the one you wrote
This is the check that finds real problems, and it takes ten seconds. Fetch your live robots.txt and diff it against the copy in your repository.
Hosting platforms and CDNs can inject directives at the edge. When that happens, the file in version control and the file crawlers read are two different documents, and nothing in a normal deployment log tells you so.
We found exactly this on our own site, and the full write-up including our initial misreading of what the injected block did is published as a study rather than buried.
From synthetic tests to real evidence
Once the synthetic run is clean, move to logs. Grep for the token, not the full string, since version numbers change under you.
Take the source IPs and verify them. Every major operator publishes either an IP range file or a reverse DNS convention, and several publish both. Bing documents a two-step check: reverse DNS on the logged IP to a name ending in search.msn.com, then a forward lookup on that name to confirm it resolves back to the same address.
Reverse DNS conventions worth knowing: Google uses crawl-***-***-***-***.googlebot.com or the geo- variant, Apple uses the applebot.apple.com domain, and Common Crawl uses crawl.commoncrawl.org.
Spoofing cuts both ways. Common Crawl warns that it is aware of crawlers falsely identifying themselves as CCBot and recommends verifying user-agent strings for authenticity. An unverified token in a log file is a claim, not a fact.
Two things not to do
Do not block by IP as a substitute for robots.txt. Anthropic states that IP blocking may not work correctly or persistently guarantee an opt-out, because it impedes the ability to read your robots.txt file in the first place. Blocking the address stops the request that would have honoured your preference.
Do not hardcode IP ranges into a permanent allowlist. Bing tells operators to refresh its list daily and separately warns against hardcoded addresses, which is a fair summary of the position across every operator publishing one of these files.
What a clean result does not buy you
Access is one link in a chain. A crawler being able to fetch a page does not mean the page was fetched, indexed, retrieved for a query, cited in an answer, or acted on by a reader. Each of those steps can fail on its own.
This test covers the first link only. It is a necessary condition and nowhere near a sufficient one, and treating a green run as evidence of AI visibility is a much bigger claim than the data supports.
# Fetch your page as a normal browser and record the byte count $ curl -sA 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)' https://example.com/ | wc -c # Now fetch the same page as OpenAI's search crawler $ curl -sA 'compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot' https://example.com/ | wc -c # A different byte count means you are serving crawlers something else. # Then check the status code, and whether your structured data survived: $ curl -sA 'compatible; OAI-SearchBot/1.0' -o /dev/null -w '%{http_code}' https://example.com/ $ curl -sA 'compatible; OAI-SearchBot/1.0' https://example.com/ | grep -c 'application/ld+json'
What is actually established, and how
Sorted by how strong the evidence is, not by how convenient it is.
| Claim | Basis |
|---|---|
| OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple and Common Crawl all publish either an IP range file, a reverse DNS convention, or both, for verifying their crawlers. | Documented by the platform |
| Bing documents a reverse DNS lookup to a name ending in search.msn.com followed by a forward lookup confirming the same IP. | Documented by the platform |
| Anthropic warns that blocking its IP addresses may not guarantee an opt-out, because it prevents its bots from reading robots.txt. | Documented by the platform |
| Common Crawl states it is aware of crawlers falsely identifying themselves as CCBot and recommends verifying user-agent strings. | Documented by the platform |
| Anthropic publishes no full user-agent strings, so a spoofing test for its bots can only use the robots.txt token. | Documented by the platform |
| Running this protocol on one live site produced 99 crawler requests that were all HTTP 200 and byte-identical to the browser control. | We measured this |
| Whether a passing access test correlates with being cited by any AI product. | Not publicly documented |
What nobody can currently tell you
Stated because the alternative is implying a certainty that does not exist.
A spoofed request measures your server, not the crawler. It cannot establish that any operator has ever fetched your site, and no method available from outside your own logs can.
Byte-length comparison catches differential serving of the initial HTML. It does not catch differences introduced by JavaScript execution, since curl runs no scripts. Common Crawl states plainly that it does not execute JavaScript, but the major AI operators do not all document their rendering behaviour.
Some bot-protection services behave differently for traffic originating from known operator address space than for a spoofed request from a random IP. That means a passing curl test can still overstate what a real crawler experiences.
We have run this protocol on one site, on one CDN, on one day. The frequency of crawler access failures across small business sites in general is unmeasured, and we are not going to estimate it from a sample of one.
Anthropic user-agent strings circulate widely in third-party writeups. We did not use them and cannot confirm them, because the operator publishes none.
What people get wrong about this
-
A 200 response means the crawler got your page.
What is actually the case
It means the crawler got a response. A truncated or stripped body returns 200 as readily as a full one, which is why byte length against a browser baseline is the real check.
-
Your robots.txt is whatever is in your repository.
What is actually the case
Edge platforms can prepend or inject directives, so the served file and the committed file can differ. Only the served file is what crawlers read.
-
Seeing a crawler token in your logs proves that crawler visited.
What is actually the case
User-agent headers are trivially spoofed by anyone. Verification requires checking the source IP against published ranges or the operator reverse DNS convention.
-
Blocking crawler IP addresses is a cleaner opt-out than robots.txt.
What is actually the case
Anthropic documents the opposite. Blocking the addresses stops the bot from reading the robots.txt that expresses your preference, so the opt-out becomes less reliable, not more.
How to check this on your own site
You should not have to take our word for any of it.
- Pick a spread of URLs, not just the homepage: the homepage, a service page, a deep content page, robots.txt, sitemap.xml, and anything rendered by JavaScript. Failures cluster on the pages nobody thinks to check.
- Record a browser baseline for each URL. Run: curl -sS -o /dev/null -w '%{http_code} %{size_download}\n' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' https://example.com/
- Repeat every URL under each crawler string. For GPTBot: curl -sS -o /dev/null -w '%{http_code} %{size_download}\n' -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' https://example.com/
- Compare the second number, not the first. A crawler response whose byte count matches the baseline passed. A 200 with a smaller body is a failure worth investigating.
- Check that structured data survived: curl -sS -A '<crawler string>' https://example.com/ | grep -c 'application/ld+json' and compare the count against the browser run.
- Diff your served robots.txt against your repository copy: curl -sS https://example.com/robots.txt | diff - ./robots.txt
- Read the served file for lines you did not write. Look for Disallow rules naming AI tokens, and for a Content-signal line, either of which may have been added by your CDN rather than by you.
- Test for rate limiting: seq 25 | xargs -P 25 -I{} curl -sS -o /dev/null -w '%{http_code}\n' -A '<crawler string>' https://example.com/ | sort | uniq -c. Any 429 or 503 in that output is a throttle a real crawler would also hit.
- Now check your access logs for the tokens themselves. Match on the substring, for example GPTBot or PerplexityBot, because published version numbers change without notice.
- Verify the logged source IPs against the operator's published range file or reverse DNS convention. A user-agent header on its own is trivially forged and proves nothing about origin.
- Re-run the whole set after any change to hosting, CDN, WAF or bot-protection settings. This is the class of change that alters crawler access silently.
Questions we get asked constantly
Do I need special tooling for this?
No. curl, diff and grep cover the whole synthetic protocol, and your existing access logs cover the verification half.
Which user-agent string should I use for ClaudeBot?
There is no published one. Anthropic documents only the robots.txt tokens, so a spoof test can send the bare token and should be reported as an approximation.
How often should this be re-run?
After every infrastructure change, and on a schedule otherwise. The configuration that breaks crawler access is usually changed by someone who was solving an unrelated problem.
What if I find a rule I did not add?
Identify which layer injected it before removing anything, then decide deliberately whether you want it. Some injected rules are training opt-outs a site owner may genuinely want to keep.
Where this came from
Every factual claim above traces to one of these. Each entry says what it supports and the date it was read, because platform documentation changes without notice.
-
OpenAI Bots — OpenAI
Published user-agent strings and per-agent IP range endpoints for GPTBot, OAI-SearchBot and ChatGPT-User.
Primary source · read 2026-09-03
-
Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
The bots.json IP list, the warning that IP blocking impedes robots.txt reads, and the absence of any published user-agent string.
Primary source · read 2026-09-03
-
Perplexity Crawlers — Perplexity
The PerplexityBot user-agent string and the published IP range endpoint used to verify its requests.
Primary source · read 2026-09-03
-
Which crawlers does Bing use? — Microsoft
The bingbot user-agent strings and the documented reverse-then-forward DNS verification against search.msn.com. Body retrieved through the help system content endpoint, since the page is a client-rendered app.
Primary source · read 2026-09-03
www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0
-
Google common crawlers — Google
Googlebot user-agent strings, the common-crawlers.json IP ranges, and the googlebot.com reverse DNS masks.
Primary source · read 2026-09-03
developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
-
CCBot — Common Crawl
The CCBot user-agent string, the ccbot.json IP ranges, the crawl.commoncrawl.org reverse DNS domain, and the warning about crawlers falsely claiming to be CCBot.
Primary source · read 2026-09-03
-
About Applebot — Apple
Applebot user-agent format and the applebot.apple.com reverse DNS domain plus the applebot.json CIDR list.
Primary source · read 2026-09-03
-
Common Crawl FAQ — Common Crawl
CCBot does not execute JavaScript and does not use cookies, which is relevant to what a curl-based test can and cannot represent.
Primary source · read 2026-09-03