Reference

How AI search actually works

Written for people who want the mechanism, not the pitch. Sourced to platform documentation, with the parts nobody has documented labelled as exactly that.

What this is

A reference section about AI search, AI crawlers and structured data. Every factual claim carries a source, the date it was read, and whether it is documented by the platform, observed in a test we ran, or inferred. Where nobody has published an answer, these pages say so instead of guessing. That is the whole editorial standard.

6 guides

AI crawlers and access

Does blocking GPTBot remove your website from ChatGPT search? OpenAI documents GPTBot and OAI-SearchBot as independent robots.txt controls. GPTBot is described as the crawler that gathers c... AI crawlers Reviewed 2026-09-03 What is the difference between an AI search crawler and an AI training crawler? A training crawler collects pages to build datasets used to train foundation models. A search crawler builds an index that AI p... AI crawlers Reviewed 2026-09-03 What is the difference between GPTBot and OAI-SearchBot? GPTBot and OAI-SearchBot are two distinct OpenAI crawlers with two distinct robots.txt tokens. OpenAI writes that GPTBot "is us... AI crawlers Reviewed 2026-09-03 What does Google-Extended actually control? Google-Extended is a robots.txt control token, not a crawler. Google states that it has no separate HTTP request user agent str... AI crawlers Reviewed 2026-09-03 How do you test whether AI crawlers can access your website? Request your own URLs with each crawler's published user-agent string and compare the result against an ordinary browser reques... AI crawlers Reviewed 2026-09-03 What does Cloudflare's AI crawler blocking actually do? Cloudflare's managed robots.txt setting generates a robots.txt file disallowing a fixed list of AI user agents. As documented o... AI crawlers Reviewed 2026-09-03

4 guides

llms.txt

What is llms.txt? llms.txt is a proposed markdown file that gives AI agents a short, curated index of links to a site's most useful pages. Jeremy... llms.txt Reviewed 2026-09-03 Do AI assistants actually read llms.txt? Some things fetch llms.txt, and they are probably not the ones site owners have in mind. An Ahrefs server-log study of 137,210 ... llms.txt Reviewed 2026-09-03 How is llms.txt different from robots.txt? robots.txt and llms.txt do opposite jobs and sit at opposite ends of the standards spectrum. robots.txt is specified by IETF RF... llms.txt Reviewed 2026-09-03 Should your website publish an llms.txt file? Publish an llms.txt if you have documentation that coding agents might navigate, because that is the one use case with measured... llms.txt Reviewed 2026-09-03

4 guides

AI search optimization

What is AI search optimization? AI search optimization is the practice of trying to influence whether a business or a page gets named, retrieved, or cited by A... AI search Reviewed 2026-09-03 How is generative engine optimization different from SEO? Generative engine optimization, or GEO, is a term from a 2024 academic paper by researchers at Princeton and IIT Delhi describi... AI search Reviewed 2026-09-03 Does schema markup help AI search? There is no good evidence that adding schema.org markup increases how often AI assistants cite a page. Google states in its doc... AI search Reviewed 2026-09-03 What is entity authority? Entity authority is an industry term for how confidently a search or AI system can identify a business as a distinct thing and ... AI search Reviewed 2026-09-03

2 guides

Local AI search

How does ChatGPT find local businesses? OpenAI discloses almost nothing. Its help documentation says only that ChatGPT may share a user location with trusted third-par... Local AI search Reviewed 2026-09-03 Do online reviews affect which businesses AI assistants recommend? Nobody has demonstrated that reviews cause an AI assistant to recommend a business. The claim traces back to a study that measu... Local AI search Reviewed 2026-09-03

Also here

Definitions, research, and worked examples

GlossaryPlain definitions of the terms this section uses. ResearchStudies we are running, with the methodology published first. Case studiesWork on real sites, including our own. Buying guidesWhat things cost and how to choose. Different question, different section.