Does llms.txt work? What it does and does not do for AI search

Last updated October 24, 2026 · Crawler facts checked October 24, 2026 against the vendors’ own documentation (sources below)

Quick answer

Not as an AI search fix, based on what the AI companies themselves publish. As of October 24, 2026, the crawler documentation from OpenAI, Anthropic, Perplexity and Google does not say their crawlers read llms.txt, and Google’s guidance says “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” llms.txt is a proposed Markdown summary of a site. It is cheap and harmless, and it can help tools that read documentation, but it does not replace crawler access in robots.txt, text in your HTML, or good answers on your pages.

What llms.txt is

It is a proposal published at llmstxt.org: a Markdown file at /llms.txt that lists a site’s most useful pages with short descriptions, so a language model can find them without parsing menus and ads. It is not a standard, and it is not a way to allow or block crawlers; robots.txt does that.

What the evidence says

Source	What it says
Google, “AI features and your website”	“You don’t need to create new machine readable files, AI text files, or markup to appear in these features. There’s also no special schema.org structured data that you need to add.”
Google’s John Mueller (via Search Engine Journal)	Said no AI services had said they use llms.txt, that server logs show they do not even check for it, and compared it to the keywords meta tag.
Google’s Gary Illyes (via Search Engine Land, July 2025)	Reported as saying Google does not support llms.txt and is not planning to.
OpenAI, Anthropic and Perplexity crawler docs	Explain crawler control through robots.txt only. None of the three pages mentions llms.txt as something their crawlers read (checked October 24, 2026).
Site owners’ logs	Some report OpenAI fetching llms.txt. A fetch is not proof that the file is used to pick sources or citations.

Note the difference between publishing and reading: OpenAI and Perplexity publish llms.txt files for their own developer docs. That tells you they find the format useful for documentation, not that their crawlers use your llms.txt to decide what to cite.

What we found on three live sites

All three sites we audited today already have an llms.txt (astorventuresllc.com 39,773 bytes, hohocam.com 8,762 bytes, kemuva.com 19,675 bytes). We keep one here too because it costs nothing. Our checker reports it but gives it 0 points, and the scores below come entirely from crawler access, HTML text, schema and page basics:

Site (home page)	Score	What the checker found
astorventuresllc.com	98/100	All 16 crawlers allowed; 3,664 words in the raw HTML. Small gaps: a 253-character meta description and no `sameAs` profile links in the schema.
hohocam.com	94/100	All 16 crawlers allowed; 2,165 words in the raw HTML; Product, Offer, FAQPage and VideoObject schema. Gaps: no `<main>` element and no published or updated date in the schema.
kemuva.com	87/100	All 16 crawlers allowed; 2,809 words in the raw HTML. Gaps: only Organization and WebSite schema on the home page, one question-style heading, no `<main>` element, no dates in the schema, and a 193-character meta description.

Checked October 24, 2026 with the free checker below, from one IP address. All three sites already let AI crawlers in, so their gaps are about structure, not access. A test page we built as a JavaScript app shell (empty <div id="root">, GPTBot blocked in robots.txt) scored 45/100 with the same checks.

What to do instead (or first)

Make sure robots.txt allows the search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot. See GPTBot vs OAI-SearchBot.
Make sure your firewall or CDN does not block them. See How to check if ChatGPT can crawl my site.
Put your main text, prices and contact details in the HTML, not only in JavaScript. See Can AI crawlers read JavaScript?
Answer the questions customers ask, in plain sentences right under a question heading.
Then, if you like, add an llms.txt. It will not hurt.

Want a second pair of eyes? The AI visibility audit checks 16 AI and search crawlers against your robots.txt, firewall and HTML, and sends a plain-English PDF report within 48 hours with ready-to-paste fixes. $79 for the report, $149 with the fixes applied. It is a diagnosis, not a promise of citations.

See the audit → Download the free checker

Frequently asked questions

Will llms.txt hurt my site?

We know of no evidence that it does. It is a plain text file that crawlers can ignore.

Is llms.txt the same as robots.txt for AI?

No. robots.txt tells crawlers what they may fetch. llms.txt only describes your content; it cannot allow or block anything.

Should I pay someone to make an llms.txt?

We would not. If you want one, it is a short Markdown list of your key pages that you can write yourself in a few minutes. Spend the effort on crawler access and on the content of your pages first.

Could this change?

Yes. If an AI company documents that its crawlers use llms.txt, the advice changes. We re-check the vendors’ documentation when we update this page.

Sources (checked October 24, 2026)

Last updated: October 24, 2026

← AI visibility audit · All guides · Home