AI Crawler Robots Policy: Who Gets to Read Your Site

There is a file on your website called robots.txt. You probably never wrote it. It may have shipped with your site builder years ago. And it is now quietly deciding whether ChatGPT, Claude, and Google's AI ever learn that your business exists.

For search engines, that file was always a courtesy. Googlebot would skip a blocked folder and keep crawling the rest. The stakes were low. With AI assistants, the same file now sits between you and the answer a buyer gets when they ask who to hire.

TL;DR

AI assistants crawl the web using named robots you can control in robots.txt. OpenAI uses GPTBot for training and OAI-SearchBot for search results. Anthropic uses ClaudeBot for training and Claude-SearchBot for search. Google uses Google-Extended to control whether your content trains Gemini. Blocking a training bot does not stop the AI from citing you from other sources, but blocking a search bot can remove you from the very answers that send buyers your way.

Key Takeaways

• AI crawlers are separate from search crawlers, and each one can be allowed or blocked independently in your robots.txt file.
• OpenAI runs GPTBot for model training and OAI-SearchBot for surfacing sites in ChatGPT search results.
• Anthropic runs ClaudeBot for training and Claude-SearchBot for search, and both respect the robots.txt Disallow rule.
• Google separates Googlebot for Search from Google-Extended, which controls whether your content trains Gemini.
• A mistaken Disallow rule can hide you from AI answers, so check your file before assuming it is fine.

What robots.txt actually controls

Robots.txt is a plain text file in your site's root folder. It tells compliant crawlers which paths they may or may not read. The rules were first described by Martijn Koster in 1994, and the format became the Robots Exclusion Protocol, now formalized in RFC 9309. The Web Robots Pages still describe it as a de facto standard that no single standards body owns.

Google spells out the mechanics in its robots.txt specification. Rules apply only to the host, protocol, and port where the file sits. A Disallow line blocks a path; an Allow line overrides it. Anything not blocked is fair game by default.

The critical detail for your business: robots.txt is a request, not a lock. A well behaved crawler follows it. A bad one does not. It was never meant to hide information. That matters because the crawlers that decide whether you get cited are well behaved, and they read this file every time.

The three OpenAI robots and what they do

OpenAI breaks its crawling into separate named robots, each with its own job. OpenAI's crawler documentation lists them with plain language on how to control each one.

OAI-SearchBot is the one that surfaces your site in ChatGPT search answers. If you block it, your pages will not appear in those search results. GPTBot is the one that crawls content for training generative models. Blocking GPTBot only stops training use. It has no effect on whether you show up when someone searches.

ChatGPT-User is the third one, and it behaves differently. It fetches a page only when a real user asks ChatGPT to read one. Because a person triggers it, robots.txt rules may not apply the same way. OpenAI says to use OAI-SearchBot, not ChatGPT-User, to manage search opt outs.

The distinction between these three robots is where most owners go wrong. The table below shows what each one does and what blocking it actually costs you.

Robot	What it does	Blocking it means
OAI-SearchBot	Surfaces your site in ChatGPT search results	You do not appear in ChatGPT search
GPTBot	Crawls content for model training	Your content is not used for training
ChatGPT-User	Fetches a page when a user asks	User-triggered reads may still happen

ClaudeBot, Claude-SearchBot, and Google-Extended

Anthropic runs the same split. Its crawl documentation names ClaudeBot for training, Claude-SearchBot for search quality, and Claude-User for user-triggered reads. Anthropic also supports the non-standard Crawl-delay extension, a polite way to slow it down without blocking it.

Google separates search from AI training too. Googlebot handles Search and always respects robots.txt. Google-Extended is the token that controls whether your content is used to train Gemini and ground its answers, and Google documents it on its crawling infrastructure page. Blocking Google-Extended does not touch your Google Search ranking. It only opts you out of Gemini training, the same way a robot rule is a convention rather than a legal requirement.

This split matters for one reason. You can say yes to being cited and no to being trained, or yes to both. They are separate switches. A blanket Disallow: / aimed at one bot can quietly turn off the switch you actually wanted on.

What to check on your own site

Open your browser and go to your domain followed by /robots.txt. Read the whole file. Most owners have not looked at it since their site launched, and many find a stray rule they never wrote.

Look for three things. First, is any AI crawler named and blocked. Second, is there a User-agent: * rule that blocks everything from unnamed bots. Third, is there a broken rule that blocks more than you intended, because a single misplaced slash in robots.txt can hide an entire folder.

If you want to be cited, allow the search bots: OAI-SearchBot, Claude-SearchBot, and Googlebot. If you do not want your content trained on, block GPTBot, ClaudeBot, and Google-Extended specifically. Those two goals coexist in the same file.

This is one piece of a larger picture. Being crawlable does not make an AI cite you. It only removes a reason the AI skipped you. The signals that actually decide who gets named are how ChatGPT picks businesses to recommend, and the entity markup we cover in what structured data means for your business.

If you are new to the whole idea, start with the five minute AI visibility primer. Your robots file is the door, not the recommendation.

The final thing to know is that all of this is a convention, not a law. NIST works on voluntary AI standards and does not mandate crawler behavior. The crawlers follow robots.txt because the platforms chose to, not because any regulator forced them to.

Sources cited in this analysis?

Frequently Asked Questions

Will blocking GPTBot stop ChatGPT from recommending my business?

No. Blocking GPTBot only stops OpenAI from using your content to train its models. It does not stop ChatGPT from citing you when a buyer searches for a recommendation. Citations depend on OAI-SearchBot and other sources, not on the training crawler.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls your content to train OpenAI's generative models, and blocking it only stops that training use. OAI-SearchBot surfaces your site in ChatGPT search results, which is what determines whether a buyer sees your recommendation. You can allow one and block the other independently.

Does a robots.txt file hide my content from AI?

Not reliably. It is a request, not a lock. Well behaved crawlers follow it, but malicious bots ignore it entirely, and robots.txt was never meant to hide information from anyone. It controls access to your pages, not secrecy for them.

What is Google-Extended?

Google-Extended is a token you set in robots.txt that controls whether your content is used to train Gemini and ground its answers. Blocking it only opts you out of AI training use. It does not affect your Google Search ranking in any way.

How do I check my own robots.txt?

Open your browser and visit your domain followed by /robots.txt. Read the whole file in full. Then look for three things: any AI crawler that is blocked, any wildcard rule that blocks everything, and any stray rule hiding more than you intended.