Does blocking GPTBot remove your website from ChatGPT search?
OpenAI publishes two separate robots.txt tokens, and only one of them governs ChatGPT search results.
Does blocking GPTBot remove your website from ChatGPT search?
OpenAI documents GPTBot and OAI-SearchBot as independent robots.txt controls. GPTBot is described as the crawler that gathers content for training foundation models. OAI-SearchBot is described as the crawler behind ChatGPT search results, and it is the token OpenAI names for search opt-outs. OpenAI's own worked example is allowing OAI-SearchBot to appear in search while disallowing GPTBot for training, so that combination is documented as supported. One honest caveat: OpenAI never prints the negative sentence, so this is documented as a supported setup rather than documented as harmless.
What OpenAI publishes about the two tokens
OpenAI’s crawler documentation puts the answer in a single paragraph near the top of the page, and the relevant sentence names this exact pairing.
That is an affirmative statement about a configuration OpenAI expects site owners to use. It names the exact pairing the question is about, and it names it as normal rather than as a workaround.
On GPTBot specifically, the same page says: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." Training is the whole stated scope of that token. Search is not mentioned in the GPTBot entry at all.
Each setting is independent of the others - for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models.OpenAI, Bots and crawlers documentation · source · read 2026-09-03
The token that does control ChatGPT search
If you want to be out of ChatGPT search answers, OAI-SearchBot is the switch. OpenAI writes: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
The navigational-link clause is worth reading twice. Opting out is documented as removal from answers, not as total disappearance from the product.
There is also a delay. OpenAI states it can take roughly 24 hours from a robots.txt update for its search systems to adjust, so a change tested ten minutes later tells you nothing.
| Token | Documented purpose | Effect of disallowing it |
|---|---|---|
| GPTBot | Crawls content that may be used to train foundation models | Signals that content should not be used in training |
| OAI-SearchBot | Surfaces websites in ChatGPT search features | Site is not shown in ChatGPT search answers |
| ChatGPT-User | Visits pages when a user asks something in ChatGPT | Not the search control; robots.txt rules may not apply |
| OAI-AdsBot | Validates safety of pages submitted as ads on ChatGPT | Only visits pages submitted as ads |
Why we phrase this as supported rather than harmless
There is a real gap between what people want OpenAI to have said and what OpenAI actually wrote. Nowhere on that page is there a sentence of the form "blocking GPTBot does not affect ChatGPT search."
What exists is the affirmative example above. Reading a clean negative out of it is a short inference and a reasonable one, but it is still an inference, and pages that present it as a direct quotation are misrepresenting the source.
So the accurate framing is: OpenAI documents allow-search-deny-training as a supported configuration. That is strong enough to act on. It is not the same as OpenAI promising no side effects.
The sentence in that paragraph most people skip
Immediately after the independence statement, OpenAI adds: "If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling."
That describes deduplication when both tokens are allowed. It says nothing about what the crawl looks like when only one is permitted.
We flag it because it is the one place in the paragraph where an operational detail could cut against the clean reading, and because extrapolating from it in either direction would be guessing.
Anthropic draws the same line, in clearer language
Anthropic runs three bots and describes them as serving model development, web search, and retrieval at a user's direction. ClaudeBot is the training one. Claude-SearchBot is the search one.
Anthropic is more explicit than OpenAI about the downside of a search block: "Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results."
Two operators, one structure. Deny the training token and the search token stays open, unless you deny that one too.
What is actually established, and how
Sorted by how strong the evidence is, not by how convenient it is.
| Claim | Basis |
|---|---|
| OpenAI documents GPTBot as the crawler for content that may be used in training its generative AI foundation models. | Documented by the platform |
| OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT search features. | Documented by the platform |
| OpenAI states that each robots.txt setting is independent, and gives allowing OAI-SearchBot while disallowing GPTBot as its own worked example. | Documented by the platform |
| Opting out of OAI-SearchBot means a site is not shown in ChatGPT search answers, though it can still appear as a navigational link. | Documented by the platform |
| OpenAI states robots.txt changes can take around 24 hours to be reflected in its search systems. | Documented by the platform |
| Disallowing GPTBot alone leaves ChatGPT search inclusion intact. This follows from the independence statement rather than from any negative sentence OpenAI prints. | Inference, not documentation |
| Whether a real site loses any measurable ChatGPT search coverage after GPTBot is disallowed. | Not publicly documented |
What nobody can currently tell you
Stated because the alternative is implying a certainty that does not exist.
OpenAI publishes no sentence stating that blocking GPTBot does not affect ChatGPT search. The negative has to be read out of an affirmative example, and we mark that as inference rather than quotation.
The crawl deduplication sentence covers the case where both tokens are allowed. What happens to crawl coverage when only OAI-SearchBot is allowed is not described anywhere we could find.
The OpenAI crawler page carries no visible last-updated date. It offers an RSS feed for changes but exposes no revision history, so there is no way to tell how long the current wording has stood.
OpenAI does not say whether content collected before a Disallow was added is removed from datasets that already exist. The wording addresses future crawling.
We have not measured this ourselves on a site that blocks GPTBot while allowing OAI-SearchBot. Everything above is documentation, not observation.
What people get wrong about this
-
Blocking GPTBot hides your business from ChatGPT.
What is actually the case
Appearance in ChatGPT search answers is governed by OAI-SearchBot. GPTBot governs training data collection, and OpenAI documents the two as independent settings.
-
Allowing GPTBot makes ChatGPT more likely to cite you.
What is actually the case
OpenAI ties GPTBot to foundation model training, not to the search index. No OpenAI document connects a GPTBot allow rule to how often a site gets cited.
-
One robots.txt line covers every OpenAI agent.
What is actually the case
Four separate tokens are published, and a rule addressed to one of them does not apply to the others. ChatGPT-User also carries a caveat that robots.txt may not apply to it.
-
A robots.txt edit shows up in ChatGPT immediately.
What is actually the case
OpenAI states the adjustment takes about a day, so any test run within minutes of the change is measuring the old state.
Questions we get asked constantly
Which robots.txt line actually removes a site from ChatGPT search?
A Disallow addressed to the OAI-SearchBot user agent. OpenAI names that token as the control for search opt-outs and says opted-out sites are not shown in ChatGPT search answers.
Does ChatGPT-User count as the search crawler?
No. OpenAI states that ChatGPT-User is not used to determine whether content may appear in Search, and directs site owners to OAI-SearchBot instead.
Is there a reason to allow GPTBot?
That is a business judgement about contributing content to model training, not a visibility question. OpenAI publishes no claim that allowing GPTBot improves search presence.
How would I confirm which OpenAI agents my site currently allows?
Fetch your own robots.txt over HTTPS and read what is served rather than what is in your repository. Those two files are not always the same.
Where this came from
Every factual claim above traces to one of these. Each entry says what it supports and the date it was read, because platform documentation changes without notice.
-
OpenAI Bots — OpenAI
GPTBot and OAI-SearchBot are documented as independent robots.txt settings, with allowing search while denying training given as the example.
Primary source · read 2026-09-03
-
OpenAI Bots: OAI-SearchBot entry — OpenAI
Sites opted out of OAI-SearchBot are not shown in ChatGPT search answers but can still appear as navigational links.
Primary source · read 2026-09-03
-
OpenAI Bots: GPTBot entry — OpenAI
Disallowing GPTBot signals that content should not be used in training generative AI foundation models.
Primary source · read 2026-09-03
-
Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic
Anthropic separates ClaudeBot for training from Claude-SearchBot for search indexing, and warns that disabling the search bot may reduce visibility.
Primary source · read 2026-09-03