How do I check if ChatGPT can crawl my website?
Quick answer
Check three things. First, your robots.txt must not disallow OAI-SearchBot, the crawler that decides whether you can appear in ChatGPT search answers (GPTBot is the separate training crawler). Second, your server or CDN must return the real page, not a 403 or a challenge page, to OpenAI’s crawlers. Third, your main text must be in the HTML itself, because OpenAI’s crawlers did not run JavaScript in Vercel’s 2024 analysis. OpenAI says robots.txt changes take about 24 hours to reach its search systems.
OpenAI’s three crawlers
| User agent | What it does (OpenAI’s words, summarized) | Obeys robots.txt? |
|---|---|---|
OAI-SearchBot | Surfaces websites in ChatGPT’s search features. Sites opted out “will not be shown in ChatGPT search answers, though can still appear as navigational links.” | Yes |
GPTBot | Crawls content that may be used to train OpenAI’s models. Blocking it signals your content should not be used for training. | Yes |
ChatGPT-User | Visits a page when a ChatGPT user asks about it. Not used for automatic crawling or to decide what appears in search. | “Robots.txt rules may not apply” because a user started the request |
Step 1: read your robots.txt
Open https://yourbusiness.com/robots.txt in a browser. Look for a group that names OAI-SearchBot or GPTBot, and for the User-agent: * group. A crawler follows only the most specific group that names it; if none does, it follows the * group. This blocks ChatGPT search for the whole site:
User-agent: OAI-SearchBot
Disallow: /So does a User-agent: * group with Disallow: / when no OpenAI crawler has its own group. If there is no robots.txt at all, every crawler is allowed.
Step 2: test what your server sends back
Request your page with OpenAI’s search crawler user agent and look at the status code. In a terminal:
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
-A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
https://yourbusiness.com/You want 200 and a size close to what a normal browser gets. A 403, 429 or 503, or a tiny response, usually means a firewall or bot-protection rule. Two caveats: this test comes from your IP, while the real crawler comes from OpenAI’s published IP list, so a CDN can treat them differently. And on Cloudflare, check Security settings → AI bot policies: Cloudflare says that from September 15, 2026, new domains block AI crawlers classed as Training or Agent on pages that show ads by default, while Search stays allowed.
Step 3: check the page without JavaScript
Use View page source (Ctrl+U or Cmd+Option+U), not Inspect. If your headline, prices and main text are not in the source, a crawler that does not run JavaScript sees an empty page. See Can AI crawlers read JavaScript websites?
Step 4: look for real visits in your logs
If your host or CDN shows raw logs or bot analytics, search for OAI-SearchBot, GPTBot and ChatGPT-User. Visits with a 200 status from IPs on OpenAI’s published lists are the strongest evidence you can get.
Or run all of this at once
The free ai_audit.py script does steps 1 to 3 for OpenAI, Anthropic, Perplexity, Google, Bing and 9 other crawlers and gives you a 0–100 score. Here is what it found on three live sites today:
| Site (home page) | Score | What the checker found |
|---|---|---|
| astorventuresllc.com | 98/100 | All 16 crawlers allowed; 3,664 words in the raw HTML. Small gaps: a 253-character meta description and no sameAs profile links in the schema. |
| hohocam.com | 94/100 | All 16 crawlers allowed; 2,165 words in the raw HTML; Product, Offer, FAQPage and VideoObject schema. Gaps: no <main> element and no published or updated date in the schema. |
| kemuva.com | 87/100 | All 16 crawlers allowed; 2,809 words in the raw HTML. Gaps: only Organization and WebSite schema on the home page, one question-style heading, no <main> element, no dates in the schema, and a 193-character meta description. |
Want a second pair of eyes? The AI visibility audit checks 16 AI and search crawlers against your robots.txt, firewall and HTML, and sends a plain-English PDF report within 48 hours with ready-to-paste fixes. $79 for the report, $149 with the fixes applied. It is a diagnosis, not a promise of citations.
Frequently asked questions
If I block GPTBot, will I disappear from ChatGPT search?
No. OpenAI says the settings are independent: you can allow OAI-SearchBot to appear in search results while disallowing GPTBot to opt out of training.
Does ChatGPT-User follow robots.txt?
OpenAI says that because ChatGPT-User visits are started by a user, robots.txt rules may not apply. It also says ChatGPT-User is not used to decide whether content appears in search; use OAI-SearchBot for that.
How long after I fix robots.txt will ChatGPT see the change?
OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust.
Can I verify that a visit really came from OpenAI?
Yes. OpenAI publishes IP lists for each crawler (openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json). User-agent strings alone can be faked.