Should you add llms.txt to your website for AI visibility?

An llms.txt file is a lightweight file that is placed in the directory of a website to help AI agents and Large Language Models (LLMs) understand, navigate, and index its content. It’s essentially a sitemap that’s curated for AI bots to tell them exactly what you want them to digest on your website. That means you can dictate which pages you would like LLMs to read.

Why use it?

There are arguments that AI search assistants find information more easily with clean, well-structured Markdown (plain text version of a page) instead of parsing through HTML (essentially the code structure and content on a page).

llms.txt files are designed to direct AI models to pages you want them to view and take information from (think product pages, key articles, services, transactional pages) to reduce the chances of AI giving incorrect or hallucinated responses about your business. 

The files help overcome the context limits by pointing models to exactly where your vital resources (such as brochures, resources, etc) live on your site and to better understand semantics across pages and content.

However, with no concrete evidence or any of the other leaders in AI (Anthropic, OpenAI, Perplexity, etc.,) coming out of the long grass and telling us whether llms.txt is beneficial, it’s left a big question mark over what we should or shouldn’t do when it comes to llms.txt.

That leaves us faced with a couple of questions: do you leave your site with no llms.txt at the risk of wasting time implementing it or detrimentally (in the future) not adding it at all? Or do you add it, and ensure your site is set up for AI visibility if it comes to fruition that, yes, actually, llms.txt does prove to be beneficial for citations and mentions in LLMs later on.

Whilst there is no ‘official’ statement as to whether llms.txt actually works or not, we’re looking at ways to implement it that are cost effective and give us, and our clients, real data to support analysis to better understand how businesses gain visibility across AI search.

We’ve recently been experimenting with the LLM Ready plugin for Craft (CMS and Commerce) websites to see if 1. AI bots are using them, and 2. if the data is beneficial.

If you have a site that’s built on Craft CMS or Commerce, John F Morton has developed LLM Ready - a plugin that makes your Craft website machine-readable by serving clean Markdown versions of your content to AI crawlers and LLMs.

Get found and cited by AI
GPTBot, ClaudeBot, PerplexityBot and other crawlers are auto-detected and served clean, structured Markdown, which may make content easier for AI tools to understand, quote and link back to when answering prompts. We’re seeing it in the plugin - AI bots are actively using llms.txt to read content.

Zero disruption
There’s no need for redesigns, template changes or content migration. The site looks identical to website visitors; it’s only AI crawlers that see the Markdown (.md) version and the pages we advise to read.

Always up to date
The Markdown (.md) is generated in real time from your existing CMS content and cached, with automatic invalidation when entries are saved. Publish once and then the page and URL is instantly available in llms.txt.

llms.txt site index
Auto-generates the emerging llms.txt standard — a sitemap for AI — so assistants can discover your most important pages.

No SEO risk
Markdown responses carry a no index header (for search engines), so  they never see duplicate content, which means your hard work across traditional SEO is untouched and unaffected. 

Future-proofing
Three detection methods (.md URLs, content negotiation, user-agent detection) cover how AI tools seemingly fetch content and where the standards are heading.

We’ve been testing the LLM Ready plugin for the last 30 days  to understand what data it shows us and how we might use that to better understand how to prepare for AI search.

Initially, the data gives us insight into how many requests are made by AI bots for the llms.txt file or the .md files created by it. 

In the last 30 days, there have been 177 requests by AI bots. We’ve only set the llms.txt to be on 3 pages (which you’ll see as the 3 content types served (homepage, about, and our services overview) and this suggests that 6 different AI bots have been on our website to read our content.

This shows us the most accessed pages by said AI bots with the most requests being for the llms.txt file and then the subsequent pages we’ve added Markdown URLs to. You’ll see that there are different request path types:

llmstxt: a request by an AI bot for the /llms.txt site index file.

Entry: a direct request from an AI bot for a specific pages Markdown version when it arrived there via the .md URL (e.g., /services/web-design.md instead of /services/web-design).

Negotiated: a direct request from an AI bot when arriving on a normal URL and asking if there is a Markdown version available instead.

Given the data, it suggests that AI bots are actively making requests for .md (Markdown) urls and and llms.txt index files, likely to speed up its process. Interestingly, it was always the homepage that was negotiated whereas pages beyond that the llms.txt file and .md URL supported AI bots. 

Perhaps this is because more often than not, homepages have a lot more code to navigate as our ‘shop windows’ and AI bots are preferring to request the Markdown alternative when available.

Direct requests tell us when an AI bot has directly accessed the Markdown file, and then we’re able to see the different AI bots that are actively requesting llms.txt or .md files. 

All in all, the data from the LLM Ready plugin suggests that, yes, AI bots are actively requesting and reading llms.txt and .md URLs to capture content more efficiently.

We only tested this on 3 pages and will be adding additions for specific articles and more resource heavy content as we continue to test. 

However, it does appear beneficial to have llms.txt and .md URLs available to AI bots and the more we understand — and the more people use AI to search — the more we expect that the facility to support AI bots with plain text content will only become more important.