Technical GEO

robots.txt for AI crawlers

robots.txt controls which AI crawlers may fetch your site. There are 20 user agents worth naming explicitly, split between crawlers that collect training data and crawlers that fetch a page live because someone just asked a question. Blocking the second group removes you from AI answers immediately. The list below is the exact set the Citely audit checks, sourced from each operator's own documentation.

The two kinds of crawler

Almost every robots.txt mistake we see comes from treating these as one thing. They have opposite consequences and opposite timescales.

Kind	What it does	Cost of blocking it
Training	Collects pages that may enter a future model.	Slow and partial. Models already trained still know you.
Retrieval	Fetches your page now, mid-answer, because a user asked.	Immediate. You cannot be cited in an answer that never read you.

Every AI crawler user agent

20 agents across 11 operators. Match these strings exactly — robots.txt user-agent tokens are matched case-insensitively but not fuzzily, so Claude-Bot matches nothing.

Operator	User agents
OpenAI	`GPTBot`, `OAI-SearchBot`, `ChatGPT-User`
Anthropic	`ClaudeBot`, `Claude-SearchBot`, `Claude-User`, `anthropic-ai`
Perplexity	`PerplexityBot`, `PerplexityUser`
Google	`Google-Extended`, `Google-CloudVertexBot`, `Google-Agent`, `Google-GeminiNotebook`
Microsoft	`Bingbot`
Apple	`Applebot-Extended`
Amazon	`Amazonbot`
Meta	`Meta-ExternalAgent`
ByteDance	`Bytespider`
Cohere	`cohere-ai`
Common Crawl	`CCBot`

Allow retrieval, decide on training

If you want to be cited, this is the shape. It permits the agents that read you live and leaves the training question to an explicit line rather than an accident.

User-Agent: *
Allow: /
Disallow: /dashboard
Disallow: /api/

User-Agent: OAI-SearchBot
User-Agent: ChatGPT-User
User-Agent: Claude-SearchBot
User-Agent: Claude-User
User-Agent: PerplexityBot
User-Agent: Perplexity-User
User-Agent: Google-Agent
User-Agent: Bingbot
Allow: /

# Content Signals: what may be done with the content, not who may fetch it.
Content-Signal: ai-train=yes, search=yes, ai-input=yes

Sitemap: https://example.com/sitemap.xml

To keep retrieval but refuse training, change the signal to ai-train=no and add a Disallow: / group for GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider. Leave the retrieval group above untouched.

Three traps

A wildcard written before these agents existed. A User-Agent: * group with Disallow: / on a staging-turned-production site blocks every crawler in the table at once, including ones added last quarter.
Assuming Google-Extended covers Google Search. It does not. Google-Extended governs Gemini training. Search indexing is Googlebot, and AI Overviews draw on the search index — you cannot opt out of one through the other.
robots.txt allowing what the page then hides. An allowed crawler that receives an empty HTML shell and a JavaScript bundle has nothing to quote. Crawler access and server-rendered content are two separate conditions, and both are scored separately in the technical criterion.

Frequently asked questions

Does blocking AI crawlers in robots.txt stop AI from citing me?

It stops the crawlers that respect robots.txt from fetching you, which removes you from the retrieval pool those assistants draw on. It does not remove what a model already learned during training, and it does not remove what other sites say about you — an assistant can still describe your product from a competitor comparison page it is allowed to read. The practical effect is that you lose the ability to be the source, while remaining a topic.

What is the difference between a training crawler and a retrieval crawler?

A training crawler collects pages that may go into a future model — GPTBot, ClaudeBot, Google-Extended, CCBot. A retrieval crawler fetches a page right now, because a user asked something and the assistant is going to read it before answering — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User. Blocking the training ones is a licensing decision with a slow effect. Blocking the retrieval ones removes you from answers immediately, and it is the mistake we see most.

Can I allow retrieval but block training?

Yes, and it is the most common considered position. Disallow the training agents (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider) and allow the retrieval ones (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bingbot). You stay citable in live answers without contributing to the next training run. Note that Google-Extended controls Gemini training but not Google Search indexing — those are separate agents and you cannot opt out of one via the other.

What is the Content-Signal directive?

A robots.txt extension from contentsignals.org that states intent rather than access: a line such as "Content-Signal: ai-train=no, search=yes, ai-input=yes" says your content may be indexed and quoted in a live answer but not used for training. It is a declaration, not an enforcement mechanism — it complements user-agent rules, it does not replace them.

Is no robots.txt at all the same as allowing everything?

Yes. A missing robots.txt means no rules, and no rules means allowed. A robots.txt that returns a 404 is treated the same way. The risk is not absence — it is a file written years ago against a wildcard rule that now catches agents nobody was thinking about when it was written.

How does the Citely audit decide a crawler is blocked?

It parses your robots.txt into user-agent groups, resolves each of the 20 agents below to its own group or the wildcard, and counts an agent as blocked only when the matched group disallows the entire site with "Disallow: /". A rule that hides one directory does not count as blocking the crawler, because it is not — you would still be citable from everywhere else. Blocked agents cost points in the technical criterion, which is 20 of the 100-point score.