Most sites have not updated robots.txt since AI crawlers became relevant. The default “Allow everything” works in 2026, but it is the result of inaction rather than deliberate decision. A real robots.txt configured for the AI era explicitly names the crawlers it allows and the ones it does not. Most sites get at least one line wrong by accident.
This is the working set of robots.txt rules for AI crawlers in 2026.
The crawlers your robots.txt should account for, organized by what they actually do.
OpenAI:
– GPTBot. Trains ChatGPT’s underlying models. Crawls broadly.
– OAI-SearchBot. Powers ChatGPT’s search index for browse mode.
– ChatGPT-User. Fires when a ChatGPT user clicks through to fetch a specific page during a conversation.
Anthropic:
– ClaudeBot. Trains Claude’s underlying models.
– Claude-SearchBot. Powers Claude.ai’s web search.
– Claude-User. Fires when a Claude user clicks through to fetch a specific page.
– anthropic-ai. Older user agent, less common in 2026 but still seen.
Perplexity:
– PerplexityBot. Trains Perplexity and powers its retrieval.
– Perplexity-User. Fires when a Perplexity user clicks through.
Google:
– Googlebot. Standard SERP crawling.
– Google-Extended. Controls whether Google uses your content for AI training (Gemini, Bard, AI Overviews learning).
– Googlebot-News, Googlebot-Image, etc. — Specific subsystems, less relevant for AI training questions.
Microsoft:
– Bingbot. Powers Bing’s index, which routes ChatGPT browse retrieval.
– MSNBot. Older naming, less common.
Meta:
– Meta-ExternalAgent. Trains Meta’s AI models.
Most WordPress and Webflow sites in 2026 ship with the default robots.txt:
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
This translates to “everyone is allowed everywhere.” Functionally, this allows all the AI crawlers above, including the training-only ones like GPTBot, ClaudeBot, and Google-Extended.
For most sites that want AI citation, this is the correct policy. The only mistake is that it is implicit. There is no record of intentional decision. If a prospect or stakeholder asks “do you allow AI training crawlers?”, the answer is “we have not blocked them, but we have not deliberately allowed them either.” That is a weaker answer than “we deliberately allow them, here is the policy.”
The right move is to make the policy explicit, even if it does not change the effective behavior.
If you want to allow all AI crawlers (the default position for sites that want to be cited inside AI answers), the explicit version looks like:
# Search engines
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# AI training and search crawlers (explicitly allowed)
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Meta-ExternalAgent
Allow: /
# Default for everything else
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
This is functionally identical to the default but makes the policy explicit. The cost is 30 extra lines in robots.txt. The benefit is that the policy is documented and visible.
If you want to allow AI search citation (so ChatGPT, Perplexity, Gemini will cite your pages) but block AI training crawlers (so OpenAI and Anthropic do not use your content to train future models), the configuration looks different.
The key distinction: most AI engines run separate crawlers for training versus search. Blocking the training crawler does not block the search crawler. The pages still get cited in chat answers because the search crawler still reaches them.
# Allow AI search crawlers (citation use)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
# Search engines (unchanged)
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
This configuration is for sites that have a reason to opt out of AI training (publishing original research, sensitive content categories, IP concerns) while still being citable in AI answers.
The trade-off: blocking GPTBot does not directly affect ChatGPT citation in 2026, because ChatGPT cites primarily from its browse retrieval (which uses Bing). But blocking training crawlers may affect long-term entity recognition as engines refresh their training data. The exact effect is hard to measure.
Three patterns that look right and break things.
Mistake one: blocking AI crawlers with a single User-agent: * Disallow: / line.
This blocks everyone, including Googlebot. The site disappears from regular search. We have audited two prospect sites where this had been deployed by mistake. Both had crashed their organic traffic and not noticed for weeks.
Mistake two: case-sensitive user agent matching.
The user agent strings are case-sensitive in robots.txt. gptbot does not match GPTBot. Sites that copy older configurations sometimes have inconsistent casing. The rule applies to the agent, not to the page paths.
Mistake three: blocking specific paths without adding new training crawlers when they emerge.
The list of AI crawlers grows quarterly as new engines launch. A robots.txt configured in 2024 missed Claude-SearchBot, Claude-User, and Meta-ExternalAgent because those did not exist yet. Sites that have not updated their explicit configuration in 12+ months are blocking some new crawlers by accident and allowing others by accident.
A quick audit:
https://yourdomain.com/robots.txt. Read the file.If you have not explicitly addressed AI crawlers in your robots.txt, decide your policy and ship it. The explicit-allow version is the default for most sites. The selective-block version is for sites with specific reasons to opt out of training.
If you want us to audit and ship the right configuration for your site, book the fit call. Productized SEO and AI search at $489 a month, no contract.
Related reading:
– What we learned in 14 days of shipping our own llms.txt
– GEO 101: Generative Engine Optimization Explained
– FAQPage schema is the single biggest AI citation move
– The new search