Which AI crawlers should I let through?
Which crawlers ChatGPT, Claude, Perplexity and Gemini use, which ones affect whether you get named in answers, and which ones are a training choice.
If you want answer engines to name you, let through the crawlers they use for search and for fetching pages when someone asks a question. Crawlers used only for model training are a separate choice, and blocking them does not remove you from search answers.
Most answer engines now run more than one crawler, each with a different job. Blocking the wrong one is one of the most common reasons a site never appears in answers.
The three kinds of AI crawler
Search crawlers read your pages so the answer engine can find and cite them when people ask questions. Block these and you are much less likely to be named.
User fetchers visit a page when a person asks the answer engine about something and it needs to read a page right then. Block these and the engine cannot read you at the moment a buyer is asking.
Training crawlers collect content that may be used to train future models. Blocking them is a decision about how your content is used, not about whether you appear in today's search answers.
The crawlers that matter for ChatGPT, Claude, Perplexity and Gemini
| Company | Name in robots.txt | What it does | Recommendation |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Finds pages for ChatGPT search answers | Allow |
| OpenAI | ChatGPT-User | Fetches a page when someone asks ChatGPT to | Allow |
| OpenAI | GPTBot | Collects content for model training | Your choice |
| Anthropic | Claude-SearchBot | Indexes pages to improve Claude's search results | Allow |
| Anthropic | Claude-User | Fetches pages when someone asks Claude a question | Allow |
| Anthropic | ClaudeBot | Collects content for model training | Your choice |
| Perplexity | PerplexityBot | Finds and links pages in Perplexity answers | Allow |
| Perplexity | Perplexity-User | Fetches pages when someone asks Perplexity | Allow |
| Googlebot | Google Search, including AI Overviews | Allow | |
| Google-Extended | Not a crawler. A setting for Gemini training and Gemini answers | Your choice, see below |
These names change from time to time, and new crawlers appear. Check each company's own crawler documentation before you make changes.
About Google-Extended
Google-Extended does not visit your site. Google crawls with its normal crawlers, and Google-Extended is a setting in robots.txt that controls how that content may be used.
It covers two things: training future Gemini models, and grounding, which is when Gemini Apps pull in content from Google's index to answer a question. Google says it does not affect whether you appear in Google Search or how you rank there.
So blocking it keeps your content out of Gemini training, and it may also make you less likely to appear in answers inside Gemini Apps. That is the trade off to weigh.
Should you block the training crawlers?
There is no single right answer, and it is your decision.
Allowing them means future models may learn about your business from your own pages, in your own words.
Blocking them keeps your content out of future training sets. Some businesses prefer this, particularly those whose content is their product.
Blocking a training crawler does not stop the same company's search crawler or user fetcher. Each one reads its own rules in robots.txt, so you can block training and still be found in search answers.
How to check what you are blocking
Open your robots.txt
Visit yourdomain.com/robots.txt in your browser.
Look for rules that name AI crawlers
Find any User-agent line with a name from the table above, followed by Disallow: /. That crawler is blocked from your whole site.
Look for a rule that blocks everything
A User-agent: * line followed by Disallow: / blocks every crawler that does not have its own rules, including answer engines.
Check your security tools
Ask whoever manages your hosting, firewall or bot protection whether AI crawlers are allowed. Some services block them by default or offer a single setting that blocks all AI bots.
robots.txt is not the only thing that can block crawlers. A firewall or bot protection service can turn them away before they ever read your robots.txt, and that block will not show up in the file at all.
An example: allow search, block training
If you want to be found in answers but keep your content out of model training, your robots.txt could include:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
The search crawlers and user fetchers do not need their own lines, as long as nothing else in the file blocks them. Ask whoever manages your website before changing robots.txt, because a mistake there can affect your normal search traffic too.
What blocking does not do
User fetchers act for a person who asked a question, so some companies say robots.txt rules may not apply to them. Blocking them in robots.txt is not a guaranteed way to keep a page out of every answer.
robots.txt is also a request, not a lock. Well known crawlers follow it. It does not protect private content. Anything that should not be public needs a login, not a robots.txt rule.
Where to start
If answer engines cannot reach your site, nothing else matters yet. Checking crawler access comes first, and it is one of the quickest fixes there is when it turns out to be the problem.