GPTBot vs OAI-SearchBot: which one should robots.txt allow?
Quick answer
They control different things. OAI-SearchBot decides whether your pages can be shown in ChatGPT search answers; GPTBot collects content that may be used to train OpenAI’s models. OpenAI says each setting is independent. If you want to appear in ChatGPT search but keep your content out of training, allow OAI-SearchBot and disallow GPTBot. Blocking both removes you from ChatGPT search answers, though OpenAI says you can still appear as a navigational link.
Which AI crawler does what
| Company | Search / answers crawler | Training crawler | User-triggered fetcher |
|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User (robots.txt “may not apply”) |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User (Anthropic says its bots honor robots.txt) |
| Perplexity | PerplexityBot (Perplexity says it is not used to train AI foundation models) | none listed | Perplexity-User (“generally ignores robots.txt rules”) |
Googlebot (Search, including its AI features) | Google-Extended token: Gemini training and grounding; does not affect Google Search | none listed here |
Ready-to-paste robots.txt blocks
Show up in AI search, stay out of training:
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yourbusiness.com/sitemap.xmlAllow everything (the default when there is no rule):
User-agent: *
Allow: /
Sitemap: https://yourbusiness.com/sitemap.xmlA common mistake: a named group cancels the * rules
Under the robots.txt standard (RFC 9309), a crawler obeys only the most specific group that names it. If you add User-agent: GPTBot with one rule, GPTBot ignores everything in your User-agent: * group, including Disallow: /admin/ or /checkout/. Repeat any private-path rules inside each named group, or list several user agents on one group as in the block above.
robots.txt is only half of it
A firewall or CDN rule can block a crawler that robots.txt allows. Cloudflare, for example, has AI bot policies that block by category (Search, Agent, Training), and says new domains get Training and Agent bots blocked on pages with ads by default from September 15, 2026. Check both, or run the free checker, which tests robots.txt and the live response for each crawler.
What we found on three live sites
| Site (home page) | Score | What the checker found |
|---|---|---|
| astorventuresllc.com | 98/100 | All 16 crawlers allowed; 3,664 words in the raw HTML. Small gaps: a 253-character meta description and no sameAs profile links in the schema. |
| hohocam.com | 94/100 | All 16 crawlers allowed; 2,165 words in the raw HTML; Product, Offer, FAQPage and VideoObject schema. Gaps: no <main> element and no published or updated date in the schema. |
| kemuva.com | 87/100 | All 16 crawlers allowed; 2,809 words in the raw HTML. Gaps: only Organization and WebSite schema on the home page, one question-style heading, no <main> element, no dates in the schema, and a 193-character meta description. |
Want a second pair of eyes? The AI visibility audit checks 16 AI and search crawlers against your robots.txt, firewall and HTML, and sends a plain-English PDF report within 48 hours with ready-to-paste fixes. $79 for the report, $149 with the fixes applied. It is a diagnosis, not a promise of citations.
Frequently asked questions
Does blocking GPTBot hurt my Google rankings?
No. Google Search uses Googlebot. GPTBot is OpenAI’s training crawler and has no role in Google Search.
Does blocking GPTBot remove my content from models already trained?
robots.txt controls future crawling. It cannot recall content collected before you added the rule.
Should a small business block AI training crawlers?
There is no single right answer. Blocking training crawlers does not, by itself, stop you from appearing in AI search answers, as long as the search crawlers are allowed. Decide based on how you feel about your content being used for training.
Does Google-Extended control AI Overviews?
No. Google says Google-Extended governs Gemini training and grounding, and does not affect inclusion or ranking in Google Search. AI Overviews are part of Google Search, which Googlebot controls.
Is “Allow: /” required?
No. Anything not disallowed is allowed. An explicit Allow line just makes your intent clear.