llms.txt: What It Is, How to Make One and Whether AI Bots Read It

An llms.txt file is a proposal, not a standard: a description of a site and a list of its most important URLs for language models and AI agents. An AI agent is a program that carries out tasks on a user's behalf, for example looking up information in documentation. Google says that its search does not use such files. On rogala.pro I have had an llms.txt file since October 8, 2026, and separately a Markdown version of every page, and I check in the server logs who fetches them. In the ten days after launch no known AI bot sent a request for llms.txt. Bots that identify themselves as ClaudeBot, OAI-SearchBot and Applebot fetched the Markdown versions of pages in that time.

What is an llms.txt file?

llms.txt is a text file in the root directory of a site that describes the site in a few lines and points to the addresses of the most important pages. It is written in Markdown, a simple text format: a # character at the start of a line makes a heading, and a link has the form [title](address). It was proposed by Jeremy Howard on September 3, 2024 (llmstxt.org). The aim was for an AI agent to quickly understand what a site consists of, without going through menus and HTML code. In August 2026 the author published a second version, based on what he had learned over two years of sites using the file. He still calls it a proposal open to comments, not an adopted standard.

llms.txt blocks nothing and does not control model training. The robots.txt file passes access rules to AI bots, but it works only on bots that respect it. Access is also restricted on the server, in the firewall and in the CDN settings, that is, a network of intermediary servers. In the post on the SEO audit I described how I check this.

How does llms.txt differ from Markdown versions of pages?

llms.txt and a Markdown version of a page are two different things, although the proposal at llmstxt.org describes both. llms.txt is one file for a whole site or part of it, that is, a list of the most important URLs with short descriptions. A Markdown version is the content of a single page in a simple format, without menus, scripts and HTML code. The links in llms.txt should lead to such versions, but each of these things works without the other (llmstxt.org).

A bot can reach a Markdown version in three ways, and the proposal at llmstxt.org describes two of them:

• At a fixed address. The proposal recommends a Markdown version at the page's address with .md added, for example page.html.md, and index.md for an address without a file name.
• Through a tag or a header. The tag <link rel="alternate" type="text/markdown"> in the page code points to the Markdown version. The tag <link rel="describedby"> points to the llms.txt file that describes this page. The second version of the proposal allows the same in the HTTP Link header, that is, in the server's response (llmstxt.org).
• Through content negotiation. A program sends the header Accept: text/markdown, that is, asks for Markdown outright, and the server at a regular address returns Markdown instead of HTML. The proposal at llmstxt.org does not describe this way. This is how, for example, a Cloudflare feature works, whose measurement I describe below.

On rogala.pro all three ways work, and llms.txt is a separate file that leads to the Markdown versions. Every page has a Markdown version with the same content at its own address with index.md added. In the page code the rel="alternate" tag points to this version, and the rel="describedby" tag points to llms.txt. In response to the Accept: text/markdown header, the server returns Markdown at the regular address. The logs show that these are two separate things: AI bots fetched the Markdown versions, and no known AI bot sent a request for llms.txt. In July 2026 I did not find a Markdown version in any of the 10 domains from my portfolio that I checked.

What does an llms.txt file look like?

The format is simple: a title, a short description and lists of links grouped under headings. The only required element is the heading with the site's name. The proposal describes the order of the elements like this (llmstxt.org):

1. A first-level heading with the name of the site or project, in Markdown # Name.
2. A short description in a blockquote, that is, a line starting with >.
3. Optional paragraphs with context, without headings.
4. Sections with a second-level heading (## Section), each with a list of links in the form - [Title](address): description.
5. By custom, a section called “Optional” with links to less important materials. In the first version an agent could skip it when the whole file did not fit in the portion of text it processes at once. The second version leaves this section only as a custom (llmstxt.org).

The simplest file for a service company has five lines:

• # Example Accounting Office
• > An accounting office in Kraków: bookkeeping for one-person businesses and companies since 2010.
• ## Offer
• - [Pricing](https://example.com/pricing/): bookkeeping prices for one-person businesses and companies
• - [Contact](https://example.com/contact/): the office's address, phone and opening hours

On rogala.pro the file has a name and description in English, a paragraph about the Markdown versions and three sections. The “Profile” section leads to the pages about me, “Blog” to the blog in Polish and English, and “Optional” to the other 25 language versions of the blog. Every link leads to a Markdown version of a page, and the description after the colon is the same text as in the page's description tag (meta description). In October 2026 the file was about 40 KB.

Some documentation sites also publish an llms-full.txt file with the whole content in one file. The proposal at llmstxt.org does not describe such a file. The llms-full.txt file in Anthropic's documentation is about 45 MB, and in Cloudflare's documentation about 69 MB (checked October 18, 2026).

What do the llms.txt files of Allegro, Empik and Morele look like?

Large Polish stores already have llms.txt, but none of the three files I checked applies the proposal in full. I downloaded the files of Allegro, Empik and Morele on October 18, 2026. I assess them against the format at llmstxt.org and against the checks that I describe in this post. All three lead to ordinary HTML pages, not to Markdown versions.

Allegro gives agents the most practical information, but part of the structure is written in bold instead of headings. The file has an update date, preferred brand names, the registered address and a contact form. It also describes the patterns of search, filter and subsequent results page addresses, so an agent can build the address of a product list itself. The guidance for models says that prices and availability have to be checked on the product page, and it gives the names of discontinued programs. Groups such as “Main categories” and “Other pages and services” have bold text instead of a ## heading. A program that reads the file according to the format will therefore assign the list of services to the section about filtering. The link description comes after a dash, not after a colon, and the 13 main categories have no descriptions.

The pages from Allegro's file returned a 403 code to my script, although the file itself downloaded without a problem. The script gave its own name and the address of my site in the User-Agent header, and I used it to check the first eight addresses from the file. I do not know whether AI bots get the same response, because only Allegro's logs show that. It is an example of a check worth doing for every file: whether the addresses from llms.txt are fetched by the same program that fetched the file.

Empik sticks to the format most faithfully, but the description of the brand reads like advertising copy. The file has a first-level heading, a description in a blockquote, ## sections and lists of links with the description after a colon. The guidance for models is specific: prices and availability should be taken from product cards, and return rules depend on the seller. The brand description contains the words “wiodąca” (leading) and “Uwielbiana marka” (beloved brand), while in this post I advise giving facts, because a model can compare the description with the site. The file has no update date. The heading and some of the labels are in English, although the rest is in Polish. Of the 35 addresses in the file, 34 returned a 200 code, and one a redirect.

Morele has a good list of categories and guides, but the file does not follow the format, so a program may fail to read it. The file starts with an invisible BOM character, and in the first line #Morele.net the space after the # is missing. In Markdown such a line is not a heading, and the heading with the name is the only required element. The file has no description in a blockquote and has several first-level headings. Not one of the 116 addresses is a Markdown link, and most are written as plain text “name - address”. Next to the categories there are lists of tags and “most frequent queries”, that is, keywords for the model, and advertising texts. Of the 116 addresses, 113 returned a 200 code, and 3 redirected to articles at new addresses.

None of the three sites gives agents Markdown versions of pages, so after reading the file an agent still fetches full HTML pages. Empik and Morele return HTML for the header Accept: text/markdown, and addresses with .md added return a 404 code. Allegro answered the same attempts with a 403 code. None of the sites has a rel="describedby" tag or a rel="alternate" type="text/markdown" tag either.

Does Google use llms.txt?

No: Google says that to be present in its search you do not need to create text files for AI or Markdown versions. Google's guide to optimizing for AI features adds that the search engine ignores such files. So they neither help visibility nor harm it (Google Search Central, updated July 10, 2026). According to Search Engine Land's report from the Search Central Live conference in July 2025, Gary Illyes said that Google does not support llms.txt (Search Engine Land).

John Mueller compared llms.txt to the keywords meta tag, that is, a tag with keywords that search engines do not use. In April 2025 he wrote on Reddit that the file is a site owner's declaration of what the site is about. An AI system can check the site itself instead (Search Engine Journal). He added that server logs show AI services not even checking whether the file exists.

Chrome and Google's search are two separate matters. Lighthouse, Chrome's tool for assessing pages, has an experimental category “Agentic Browsing” that checks, among other things, whether there is an llms.txt in the root directory of the domain (Chrome for Developers). This check assesses how well a site is prepared for agents in the browser. Its result says nothing about positions in Google's search.

Do AI bots read llms.txt?

The available data indicate that AI search engine bots rarely fetch llms.txt, while GPTBot and Claude Code, a developer assistant, do so more often. The three studies that I checked in their original publications measure different things: the llms.txt file itself, Markdown versions sent through the Accept header, and separate Markdown addresses.

• Ahrefs analyzed 137,210 domains from its traffic analytics in May 2026. 28% of them had llms.txt, and up to 97% of those files received not a single request in May. 96% of the requests that reached the files were sent by bots. AI search engine bots, for example OAI-SearchBot and PerplexityBot, sent a combined 233 requests, 1.1% of all. Among named AI tools GPTBot sent the most requests, and Claude Code was second. GPTBot is OpenAI's bot that collects data for models, and Claude Code is a developer assistant from Anthropic (Ahrefs, June 15, 2026).
• Suganthan Mohanadasan recorded requests with the header Accept: text/markdown for 44 days after switching on a Cloudflare feature that serves Markdown to agents. The author gives 1,421 requests (his table adds up to 1,373). 639 of them were sent by a headless browser, and 500 by Anthropic's infrastructure (according to the author not ClaudeBot). GPTBot and PerplexityBot are not in the table (Suganthan Mohanadasan, April 19, 2026).
• Otterly compared HTML pages with their Markdown versions at separate addresses, linked in the footer, for 14 days. The HTML pages had 137 visits from AI bots, and the Markdown versions zero. No AI platform cited a Markdown address (Otterly, updated April 1, 2026).

The Ahrefs study measures requests, not what the bot did with the file. That GPTBot fetches llms.txt does not mean the file changes ChatGPT's answers. Ahrefs also notes that its customers are more technical and SEO-aware than the average site. Across the whole web the share of sites with llms.txt is therefore probably lower than 28%.

What did my site's logs show after I deployed llms.txt?

For ten days after launch no known AI bot sent a request for llms.txt, and the Markdown versions were fetched by bots that did not ask for that file. On October 8, 2026 I published on rogala.pro an llms.txt and a Markdown version of every page (114 addresses on the day of launch). I download the access logs from the hosting and count three kinds of requests: for llms.txt, for index.md files, that is, Markdown versions at each page's own address, and for Markdown responses at a page's regular address.

I recognize Markdown responses at a regular address indirectly, so their count is an approximation. The hosting does not record the Accept header, so I look for responses whose size equals one of the known sizes of the Markdown versions. After a content change I do not know the old sizes, so I may have missed some of these responses. A chance match of size is possible, but unlikely with sizes counted to the byte.

Results for October 8-17, 2026, after subtracting my own requests:

• I recorded 5 requests for llms.txt, of which 3 ended with the file being downloaded. Once it was downloaded by an SEO tool's bot, and twice by anonymous scanners posing as a phone browser. No known AI bot sent a request for this file.
• Requests for index.md files were sent directly by ClaudeBot (123), OAI-SearchBot (35) and Applebot (27). ClaudeBot is Anthropic's indexing bot, a different program from the Claude Code in the Ahrefs study. The hosting rejected 34 ClaudeBot requests with a 429 code, that is, “too many requests”, so 89 ClaudeBot requests ended with the file being downloaded.
• I detected Markdown responses at a regular address only in 22 requests from ExaSearchBot, mainly on the home page. ExaSearchBot is a search engine bot for AI applications.
• Googlebot sent 169 requests for HTML pages, and I did not detect Markdown responses for it. ChatGPT-User, which fetches pages at the request of a ChatGPT user, sent 8 requests for HTML pages.

The logs do not show how the bots knew the addresses of the Markdown versions. In that time none of them sent a request for llms.txt. The <link rel="alternate"> tag on the HTML pages gives the address of the Markdown version, so that is a possible route, but not the only one. A fetch also does not say whether the bot used that version in an answer, and that needs a separate measurement.

Your own tests are easy to mistake for bot traffic. When I checked the deployment on October 17, my script fetched Markdown files with Googlebot's identifier, which produced 115 requests from “Googlebot” from my IP address. Without subtracting my own addresses, the logs would have shown Googlebot fetching Markdown, although I did not detect the real Googlebot fetching Markdown.

When is it worth making llms.txt?

llms.txt makes the most sense where developer assistants read the site: documentation, APIs and developer tools. An API is an interface through which programs exchange data. In the Ahrefs study Claude Code was second after GPTBot among AI tools by the number of requests for the file. An assistant in a code editor that gets the address of documentation can reach the right page faster from a list of links than through navigation.

For a store or a service company, llms.txt is a cheap add-on, not a way to get AI traffic. I know of no evidence that the file increases traffic or citations. Before you make it, check whether AI bots can fetch your site at all. Then check whether the content is visible in HTML without running JavaScript. I described both checks in the post on AI SEO and GEO.

The version for bots must have the same content as the page for people, and only the format may differ. Google describes cloaking, that is, showing a search engine different content than users, in its spam policies (Google Search Central). A Markdown version with the same information, prices and terms is another format of the same page, while a version with different content or different terms is a risk. On rogala.pro the Markdown version is chosen by the Accept header, not by the bot's name. In the Link header the Markdown version points to the HTML page as the canonical URL, that is, the preferred version of the page.

How do you make an llms.txt file?

You can make the file yourself if you know which pages are important:

1. Choose the pages that someone asking about your company or product should know: the offer, pricing, documentation, contact, the most important texts. Do not paste in the whole sitemap.
2. Write a heading with the name and one or two sentences of description in a blockquote. The description should give facts about the company, not an advertising slogan, because a model can compare it with the site.
3. Group the links under second-level headings and next to each write one sentence about what is on the page.
4. Save the file as /llms.txt in the root directory of the domain, in UTF-8 encoding. Check that the address returns a 200 code, that is, a correct response, and plain text, not an HTML page, which some systems show at every unknown address. If your system does not let you upload a file to the root directory, ask your store or hosting provider.
5. Generate the file from the same source as the site, so that the information in the file matches the content. On rogala.pro llms.txt is created at every build of the site from the same data as the HTML.

Is an llms.txt generator enough?

A generator will prepare a first version of the file, but the choice of pages and the descriptions must be checked by hand. According to DataForSEO, the keyword “llms txt generator” is searched about 70 times a month in Poland, and “llms.txt generator” about 480 times in the USA (the median over 12 months). If a generator takes addresses from the sitemap or from a bot crawl of the site, it does not know which pages are important to you. Before publishing, check five things:

• whether the file contains only the pages you want to show, without login pages, the cart and pages under construction;
• whether every address works and returns a 200 code;
• whether the descriptions match the current content of the pages, the prices and the offer;
• whether the file contains no information that is not on the site;
• how the file will be updated after changes on the site.

How do you check whether bots fetch llms.txt?

The answer is in the server logs, not in AI visibility tools. Download the raw access logs from the hosting panel and search them for the address /llms.txt and the addresses of the Markdown versions. For every request the log records the IP address, the date, the response code, the size and the name of the program that sent the request, that is, the User-Agent.

A bot's name can be forged, so I verify the more important requests by IP address. OpenAI publishes lists of its bots' addresses, and in my logs 3 of the 4 address blocks of OAI-SearchBot were on such a list. For the same reason I subtract my own tests, because my check script posed as Googlebot.

Download the logs regularly, because hosting does not keep them for long. My hosting keeps logs for 14 days, so to have a month of measurement I download the archives every few days. I measure the brand's presence in AI answers separately. I described that measurement in the post on brand visibility in AI.

Frequently asked questions about llms.txt

Where to start?

Start with bot access and add llms.txt last. Open /robots.txt and check the rules for AI bots. Then download the logs from recent days and see which bots visit the site and with what response code. If you have documentation or a product for developers, add llms.txt and Markdown versions right away. For a store or a service company, llms.txt is an add-on after these two checks. If you want to do it together, write through the contact page.