Quick answer: AI companies use separate crawlers for training, search and user-triggered visits, so one blanket rule is a mistake. OpenAI says GPTBot is for model training, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User fetches pages when a person asks, where robots.txt rules may not apply. Google-Extended is only a robots.txt token for Gemini use and does not affect Google Search. Allow search crawlers if you want to be cited, and decide on training separately.
Key takeaways
- Training, search and user-initiated fetching are different things. Control them separately.
- Blocking OAI-SearchBot removes your site from ChatGPT search answers, according to OpenAI.
- Blocking GPTBot signals that your content should not be used to train OpenAI's models.
- Google-Extended does not affect inclusion or ranking in Google Search. Googlebot handles Search.
- robots.txt is a request, not security. Protect private content with logins.
- Never block Googlebot or Bingbot by accident when adding AI rules.
Customers now ask ChatGPT, Gemini and other assistants for businesses and services. Whether your pages appear in those answers depends partly on which crawlers you let in. Here is how to decide.
Table of Contents
Which AI crawlers matter and what do they do?
| Crawler | Purpose (per its publisher) | Effect of blocking |
|---|---|---|
| GPTBot (OpenAI) | Crawls content that may be used to train OpenAI's generative AI models | Signals your content should not be used for training |
| OAI-SearchBot (OpenAI) | Surfaces websites in ChatGPT's search features | Your site is not shown in ChatGPT search answers. OpenAI says changes can take about 24 hours. |
| ChatGPT-User (OpenAI) | Visits pages when a user asks ChatGPT or a Custom GPT to | OpenAI says robots.txt rules may not apply to user-initiated actions |
| Google-Extended (Google) | A robots.txt token controlling use of Google-crawled content for Gemini training and grounding | Does not affect Google Search crawling, indexing or ranking |
Other AI companies publish their own crawler names. Check each company's documentation before adding rules.
How should you decide?
- Want to be cited in AI answers? Allow search crawlers such as OAI-SearchBot and, of course, Googlebot and Bingbot.
- Worried about training use? That is a business and legal choice. Blocking training tokens like GPTBot or Google-Extended states your preference.
- Selling original content? Publishers often block training crawlers but allow search ones.
- Typical service business? Visibility usually matters more than training opt-out, so allowing both is common. Decide deliberately.
What does a robots.txt example look like?
# Allow normal search crawlers
User-agent: Googlebot
Allow: /
User-agent: OAI-SearchBot
Allow: /
# Opt out of model training only
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Everyone else
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlThis is an example, not advice for every site. Keep the sitemap line, and test the file in Search Console before and after.
What about Content-Signal and llms.txt?
Content-Signal lines in robots.txt are an emerging way to state preferences such as search, AI input and AI training. They are not an official standard and not every crawler honours them. Google says files like llms.txt are not needed to appear in Google Search, so treat them as optional extras. Our GEO guide explains what actually helps you get cited.
How do you check your setup?
- Open yoursite.com/robots.txt and read it line by line.
- Confirm no broad
Disallow: /rule applies to User-agent: *. - Use URL Inspection in Search Console to confirm Google can crawl key pages.
- Check your server logs for crawler names to see who visits.
- Review the file whenever you add a plugin, CDN or firewall rule that touches bots.
Frequently Asked Questions
Should I block GPTBot?
It depends. Blocking GPTBot signals that your content should not be used to train OpenAI's models. It does not remove you from ChatGPT search, which uses OAI-SearchBot.
Will blocking Google-Extended hurt my SEO?
No. Google-Extended only controls use of content for Gemini training and grounding. It does not affect inclusion or ranking in Google Search.
What is OAI-SearchBot?
OpenAI's crawler that surfaces websites in ChatGPT's search features. If you block it, your site will not be shown in ChatGPT search answers.
Does robots.txt stop ChatGPT-User?
OpenAI says that because ChatGPT-User visits are initiated by a user, robots.txt rules may not apply.
Is robots.txt a security tool?
No. It is a request that well-behaved crawlers follow. Protect private content with authentication.
Can you set up robots.txt and sitemaps for my site?
Yes. We configure robots.txt, sitemaps, canonicals and crawl settings as part of our technical SEO service.
Sources and further reading
- OpenAI: Overview of OpenAI crawlers
- Google Search Central: Google's common crawlers
- Google Search Central: AI features and your website
Crawler behaviour is based on OpenAI's documentation and a summary of Google's crawler documentation as of November 2026 and may change.