llms.txt files fail discovery: Mueller's data

John Mueller states llms.txt cannot help LLMs choose which sites to display. SE Ranking's analysis of 300,000 domains confirms there is no statistical link between adopting these files and citation frequency in model outputs. Mueller reinforced this on the Search Off the Record podcast, explaining that systems cannot trust a file where every site claims to be the best.

The discussion reveals why architectural limitations prevent LLMs from using such files for differentiation. Just as the keywords meta tag failed when every publisher stuffed them, universal self-promotion renders the signal noise. Mueller noted that while these files fail at discovery, they may assist agents already navigating a specific site to complete tasks like purchasing a photograph.

Readers will learn why HTML pages remain the only reliable foundation for crawling and how internal links drive actual visibility. Finally, the text explores why converting content to Markdown solely for bots represents a misallocation of resources according to Google leadership.

The Role of Self-Reported Signals in LLM Discovery

Why John Mueller Says llms.txt Fails as a Discovery Signal

Self-reported data lacks the objective differentiation required for LLM discovery. During a Search Off the Record episode, John Mueller argued that LLM systems cannot trust these files for ranking since every site claims superiority, rendering the signal useless for selection. He compared this vulnerability to the defunct keywords meta tag, noting that self-declared signals offer no way to differentiate between websites. Mueller directed attention back to standard HTML pages and internal links for the purpose of discovery. The core failure mode is not gaming but the absence of a comparative mechanism. A researcher reviewed over 1,400 individual llms.txt files and their corresponding performance metrics on Google Search in November 2025 to evaluate their efficacy. Even accurate files provide no basis for an AI to prefer one source over another when both assert equal relevance. This creates a deadlock where self-reported signals offer no lift for visibility. However, Mueller carved out a narrow operational role once an agent has already selected a site. The file functions as a local directory for navigation rather than a global beacon for discovery. Relying on standard HTML and internal links remains the primary method for initial indexing. The strategic implication is clear: operators should optimize internal linking structures instead of maintaining parallel llms.txt files for search visibility.

Real-World Impact of llms.txt on LLM Bot Traffic and Citations

Empirical data suggests llms.txt files fail to influence LLM discovery or increase citation frequency in production environments. Analysis of 300,000 domains found no statistical link between file adoption and answer visibility, confirming the signal carries no weight for differentiation. Traffic logs reveal a mechanistic reason: substantial citation drivers rarely interact with the file during the selection phase. An analysis by Limy.ai covering 515,382,577 LLM bot traffic events revealed that the share of requests from substantial citation-driving bots actually touching the /llms.txt endpoint is statistically negligible. Unlike standard HTML, which provides verifiable context through internal linking structures, self-reported manifests offer no basis for an LLM to rank one site above another when every competitor submits identical assertions. The file functions best as an on-page navigation aid for agents already inside the domain, not as an entry-point signal. Relying on strong HTML architectures remains the most reliable method for ensuring visibility.

Architectural Limits vs Signal Saturation in LLM File Processing

The phrase "by design" in Mueller's argument signals that self-reported signals lack the external validation required for LLM discovery differentiation. This constraint stems from an inability to trust site-owner claims without independent verification mechanisms. Mueller explicitly compared this vulnerability to the defunct keywords meta tag, indicating that such inputs hold no value for ranking algorithms when universally adopted. The mechanism is simple: if every site claims supremacy, the signal provides zero differentiation value for a system selecting sources. Evidence confirms that LLM bot traffic largely ignores these files during the selection phase. While some operators hope for future utility, the signal saturation problem mirrors historical failures where universal adoption destroyed metric value. One interpretation suggests LLM systems are architected to evaluate web content rather than self-reported files for source selection, whereas a signal-saturation view implies the data exists but is discarded as noise. The cost of misinterpreting this distinction is wasted engineering effort on files that function only as local store directories, not global maps.

Operators must recognize that internal linking remains the sole reliable driver for initial site selection. The implication for GEO strategy is clear: optimize HTML structures rather than relying on unverified manifest files. Prioritizing standard crawling pathways over experimental file types is advisable until agentic standards mature beyond self-declaration models.

Architectural Limitations of LLM Crawling Infrastructure

Why LLM Systems Treat Self-Reported llms.txt as Untrustworthy by Design

Self-declared data offers no objective basis for differentiating site quality, so LLM architectures discard llms.txt for discovery. Google's John Mueller called building Markdown pages specifically for bots "a stupid idea" while comparing the llms.txt format to the obsolete keywords meta tag. Universal inflation rendered that historical signal useless for sorting, a fate shared by this modern equivalent. Site owners face no restrictions on adding self-serving content, making the format inherently untrustworthy. Operators face a hard constraint on automation strategies because HTML crawling remains the primary path for initial visibility.

Attempts to use these files for ranking ignore the fundamental requirement of external verification. Even accurate files fail to help an agent choose between two valid competitors. The limitation is not merely about potential abuse but the absence of a comparative mechanism within the file format itself. Publishers must prioritize standard HTML structures over auxiliary markdown files so crawlers can parse and evaluate content relationships effectively. Relying on self-reported manifests for discovery assumes a level of systemic trust that current LLM infrastructure explicitly rejects by design.

Real-World Evidence: Zero Correlation Between llms.txt Adoption and LLM Citations

Production logs confirm that LLM bot traffic from substantial citation drivers rarely accesses /llms.txt during the discovery phase. An analysis covering a massive volume of bot events reveals that requests touching this file are statistically negligible, proving crawlers do not consult it to select sources. This behavioral data aligns with broader market findings where no statistical link was observed between file adoption and answer visibility across 300,000 domains. The absence of correlation demonstrates that self-declared signals fail to influence which sites an LLM chooses to quote.

The fundamental failure lies in the inability of self-reported signals to provide comparative value. When every operator claims their content is superior, the LLM system lacks a trusted mechanism to differentiate one site from another based solely on the file's contents. Fixing low citation rates requires optimizing standard HTML pages and internal linking structures rather than maintaining separate bot directives. The file serves only as a local directory for agents already navigating a specific domain, not as a global discovery beacon.

The Differentiation Failure: Why Self-Reported Files Cannot Replace HTML Discovery

Self-reported llms.txt files fail as discovery signals because LLM architectures cannot trust unverified claims to differentiate between competing websites. When every operator asserts they host the "best website ever," the system lacks an objective mechanism to rank one site above another without external validation. This structural flaw mirrors the historical collapse of meta keywords, where universal inflation rendered the signal useless for sorting quality content. The format remains inherently untrustworthy precisely because site owners face no restrictions on adding self-serving content. Production logs confirm that LLM bot traffic from substantial citation drivers rarely accesses these files during the selection phase. Deploying a llms.txt file provides no competitive advantage in ranking algorithms that prioritize HTML discovery paths.

Addressing manipulation through penalties does not solve the issue of how self-reported files assist an LLM in choosing between competitors. The cost of maintaining these files is time spent on a signal that offers zero differentiation value. Engineering resources are improved spent strengthening internal linking structures rather than generating Markdown files that bots ignore.

Strategic Application of Structured Instructions for AI Agents

Defining the Navigation-Only Scope of llms.txt for AI Agents

llms.txt files only matter once an automated visitor reaches a specific domain. This narrow window applies when a system sits on a website, using the file as a local map instead of a billboard. Navigation differs sharply from discovery, restricting the file's role to operational guidance inside a known environment. An agent trying to purchase a photograph from a specific site illustrates this utility perfectly; the LLM visits the location and hunts for instructions to find necessary resources. Attempts at using these instructions for initial site selection fail because LLMs cannot trust self-reported signals to differentiate between competitors. Operators should deploy these files solely to structure internal pathways for agents that have already arrived.

Standardizing such interactions feels premature while formats like WebMCP remain under discussion alongside other file types without becoming a standard. Industry consensus suggests agentic systems could take significant time before settling on a universal protocol. Organizations should treat llms.txt as a functional utility for task completion rather than a lever for search ranking or citation frequency.

Implementing llms.txt for Automated Purchase Agents

Operators configure llms.txt to guide transaction flows only after an agent lands on the domain. This file functions as a local directory for a visitor already inside the store, mapping paths to checkout endpoints rather than attracting initial traffic. In the example of an agent purchasing a photograph, the system consults the file to locate the specific cart integration and payment gateway URLs required to finish the job. Production data confirms that substantial citation-driving bots rarely request this file during discovery, rendering it ineffective for visibility. Operational value exists strictly in reducing friction for known visitors executing specific tasks like buying images or booking services. No universal standard for these navigation instructions currently exists across the industry. Protocols such as WebMCP remain under discussion, with no single format yet established as the industry.

Teams building automated purchase flows today must treat any current specification as provisional. Maintaining strong standard HTML links alongside any experimental text files guarantees agents can always find the checkout page regardless of file format changes.

Organizations should limit llms.txt deployment to specific use cases where an agent is already known to be on-site, rather than relying on it for broad discovery.

Risks of Relying on Unsettled Agent Navigation Standards

Investing in WebMCP or similar proto-standards exposes operations to uncertainty before formats stabilize. Agent navigation protocols remain unsettled, with various file types currently under discussion and none having achieved standard status. Premature implementation locks teams into maintenance cycles for specifications that may fundamentally change or disappear entirely. The primary failure mode involves allocating engineering resources to llms.txt files that substantial bots ignore during the discovery phase.

Conflating navigation aids with discovery mechanisms creates strategic errors. A file might guide an agent already present on a domain, yet it cannot influence which site the system chooses to visit first. Enterprises should prioritize standard HTML structures, as HTML remains the foundation for crawling and discovery. Strong internal linking remains the most reliable method for ensuring visibility until agent behaviors stabilize.

Optimizing Site Architecture for Reliable AI Visibility

Implementation: Defining the Navigation-Only Scope of llms.txt for AI Agents

The llms.txt specification functions strictly as an intra-site navigational aid rather than a cross-domain discovery signal. This boundary exists because the file only helps an automated system once that system is already present on a website to locate helpful resources. Unlike robots.txt, which blocks crawlers universally, this file offers no mechanism for ranking one domain against competitors. Operators should restrict implementation to guiding post-discovery agent tasks:

1. Map internal purchase flows for autonomous buying agents.
2. Define API endpoints for data extraction within the domain.
3. Exclude broad marketing claims that lack external verification.

Maintenance overhead often outweighs actual utility for large enterprises managing 10,000+ pages. Synchronizing this file represents a recurring operational expense with no verified benefit for initial visibility. Because standards like WebMCP remain unsettled, maintaining these files requires ongoing developer resources without guaranteed long-term stability. Experts advise treating these files as temporary helpers for specific agent workflows instead of core SEO infrastructure. The architecture supports known visitors, not new ones.

Implementation: Implementing llms.txt for Automated Purchase Agents

Deploy llms.txt to guide transactional logic for agents already within your domain boundaries. This file operates as a local manifest, directing autonomous systems to specific checkout endpoints rather than influencing external selection. This utility is illustrated by an agent purchasing a photograph where the system consults the file to locate the exact cart integration URL required to finalize the sale. Unlike discovery signals, this approach functions like a store directory for a customer who has already entered the building. Production environments confirm that substantial citation-driving bots rarely request this file during initial discovery, rendering it ineffective for visibility gains. The share of requests touching the file remains statistically negligible among agents like GPTBot or ClaudeBot. Consequently, operators must treat this specification as a navigation aid exclusively for post-discovery interactions.

1. Define purchase flows explicitly for autonomous buying agents.
2. Map API endpoints for internal data extraction tasks.
3. Exclude any self-promotional content intended for ranking differentiation.

A critical tension exists between immediate implementation and protocol stability. Standards for agent navigation remain unsettled, with consensus potentially requiring six months to a year, or longer, to emerge. Allocating engineering resources to specific WebMCP configurations now may require significant rework as formats stabilize. The operational consequence is clear: teams should limit deployment to high-value transactional paths where the ROI justifies maintaining a volatile specification. Do not expect this file to solve problems related to AI not choosing your site for initial queries. Experts recommend restricting llms.txt usage to known internal workflows until broader standards consolidate.

Implementation: Risks of Relying on Unsettled Agent Navigation Standards

Investment in WebMCP or similar proto-standards requires careful consideration of timeline risks. Industry estimates suggest that consensus on agent navigation formats will not emerge for six months to a year, or longer, leaving early adopters managing volatile specifications. The primary failure mode involves allocating scarce developer cycles to llms.txt maintenance that yields no discovery benefit. SE Ranking's analysis of 300,000 domains found no statistical link between file adoption and citation frequency in LLM answers, confirming that self-reported signals do not drive visibility. Operators must prioritize stable HTML architectures over experimental file types while standards mature:

1. Defer implementation of non-standard agent files until specifications stabilize.
2. Audit current internal linking structures to ensure strong crawler access.
3. Monitor industry developments without committing production resources to drafts.

The hidden cost of chasing these unsettled formats is the opportunity cost of neglecting proven site architecture fundamentals. While teams tweak experimental pointers, competitors strengthen the organic signals that actually influence model selection. Experts recommend focusing exclusively on optimizing existing content structures rather than betting on unproven navigation aids. The zero effect of these files on rankings further validates a wait-and-see approach. Until the system converges, standard crawling remains the only reliable path for AI visibility.