AI citation share: why 97% of llms.txt files fail

A mere 1% of fetches for llms.txt files come from citation-generating bots, according to Ahrefs data. This stark reality defines the current state of AI citation: self-reported files fail to drive visibility while Microsoft attempts to quantify influence through new dashboard metrics. The industry is shifting from hoping for inclusion to measuring actual grounding query performance, though the available data remains fragmented across competing ecosystems.

Citation Share in Bing Webmaster Tools offers the first look at competitive AI visibility, despite being limited to Copilot and Bing answers. We must also examine the architecture of agentic resource discovery, analyzing why Google and coalitions are publishing agent specifications while llms.txt adoption stalls among substantial retrieval models. Finally, we compare the emerging toolsets for tracking these metrics across the Bing and Google ecosystems to identify where real measurement is possible.

Structured files alone cannot differentiate your site when Search Console offers no citation counts. As Matt G. Southern notes, the focus must shift to how Intents and Topics group queries within these new dashboards. Enterium provides the necessary framework to interpret these disjointed signals without relying on incomplete third-party previews.

The Role of AI Citation Share and Structured Files in Modern Search

Defining AI Citation Share and the Structured-File Ask

AI citation share quantifies the percentage of grounded AI responses attributing a specific domain for a given query. Microsoft defines this metric within Bing Webmaster Tools to measure relative visibility against competitors rather than raw traffic volume. The structured-file ask refers to the industry-wide request for domains to publish machine-readable manifests like `llms.txt` on their root paths. These files intend to instruct retrieval agents on content availability, yet Google's John Mueller argues the format cannot help LLM systems differentiate between websites for discovery. Self-reported data lacks the authority to alter ranking logic without independent verification signals.

Empirical analysis supports this limitation across large-scale deployments. Ahrefs data covering 137,000 domains revealed that 97% of `llms.txt` files received zero requests from crawlers. The retrieval bots responsible for generating citations, such as ChatGPT and Perplexity, accounted for just 1% of the fetches that did occur. This disparity indicates that publishing a structured file does not guarantee agent interaction or improved citation frequency.

Organizations should treat `llms.txt` as a low-cost adjunct for specific coding agents rather than a primary lever for search visibility. Relying on self-declared files to drive discovery ignores the reality that substantial models prioritize established HTML structures and internal linking patterns. Visibility depends on whether bots read the file, not merely on its existence.

Microsoft AI Citation Tools and Real-World Measurement

Organizations should publish llms.txt only after validating that their target AI agents actually retrieve the file. Microsoft recently deployed Citation Share metrics within Bing Webmaster Tools to measure this exact attribution rate against competitor domains for specific grounding queries. This utility tracks whether publishing structured manifests results in tangible visibility gains across Copilot and Bing Answers. However, relying solely on file publication ignores the retrieval behavior of the agents themselves. Data indicates that while many domains host these files, the bots capable of generating citations rarely request them, with retrieval agents accounting for a minimal fraction of total fetches. The limitation is that Citation Share currently isolates Microsoft's system, offering no visibility into Google's ranking logic where similar citation counts remain absent from Search Console. Consequently, a domain might optimize for Bing's parser while remaining invisible to other substantial models that disregard the format entirely. Teams must verify agent activity before committing resources to extensive manifest maintenance. Measure the citation delta directly rather than assuming presence equals performance.

The Risk of Unproven Adoption for llms.txt and OKF Formats

Google's John Mueller confirms llms.txt files lack mechanisms to help LLMs differentiate sites for discovery. This technical constraint means self-reported manifests cannot influence which website an agent surfaces for a query. Newer specifications like Open Knowledge Format and Agentic Resource Discovery repeat this structural request without solving the fundamental adoption gap. These formats function similarly to a sitemap but target agent capabilities rather than page URLs. However, widespread utility remains theoretical while substantial retrieval bots rarely fetch these resources. Early data indicates citation-generating agents account for a negligible fraction of file requests, leaving most deployments invisible to the very tools they intend to serve. The cost of premature implementation is measurable engineering time spent maintaining files that provide no ranking signal. Enterprises risk investing in OKF pipelines that yield zero visibility returns until agent behavior shifts. Enterium advises clients to treat these specifications as experimental artifacts rather than production requirements. Teams should prioritize monitoring actual citation share metrics over publishing unverified file types. Waiting for proven agent retrieval patterns reduces waste and aligns infrastructure with observed bot behavior.

Inside the Architecture of OKF and Agentic Resource Discovery

OKF Markdown Packaging and ARD Draft Specifications

Open Knowledge Format acts as a markdown wrapper bundling datasets and runbooks specifically for agent consumption, currently resting at version 0.1. This structure permits AI systems to digest organizational knowledge without depending on HTML heuristics. Agentic Resource Discovery functions differently as a draft protocol defining how agents locate and verify tools. A coalition pushes this standard toward version 0.9, aiming to help agents seek verified capabilities instead of merely indexing pages. Both specifications apply structured files hosted on a domain, mirroring the approach taken by llms.txt.

Parsing Structured Files for AI Agent Tool Discovery

The Open Knowledge Format wraps datasets in markdown, allowing agents to ingest organizational knowledge without relying on visual heuristics. Agentic Resource Discovery defines how agents verify tools before execution. This shift represents a sitemap reborn for capabilities rather than pages. The mechanism differs fundamentally from legacy indexing. Traditional crawlers traverse links to build a search index, whereas agentic systems query these structured files to determine immediate actionability.

A tension exists because a self-reported file cannot make an LLM choose a site, and bots generating citations barely fetch such files. These files remain inert assets without widespread agent implementation. Operators must treat these specifications as optional capability announcements rather than mandatory discovery levers. Self-reported files cannot differentiate sites for discovery, a constraint applying equally to OKF and ARD until agent behavior shifts. Enterprises should maintain these files for future compatibility but prioritize direct API integrations for immediate agent utility.

Unproven Adoption Risks in OKF Version 0.1 and ARD 0.9

Deploying Open Knowledge Format version 0.1 immediately introduces strategic considerations due to unproven bot engagement. The specification remains an early draft, lacking the system validation required for production reliance. Agentic Resource Discovery at version 0.9 functions as a provisional framework rather than a stable standard. A coalition drives this initiative, yet broad agent adoption remains theoretical.

Maintaining parallel structures for evolving specs costs more than current visibility gains justify. Structured files cannot force agent interaction without upstream software support. Teams should monitor these specifications within isolated staging environments. Verifying actual crawler user-agent strings before altering production domain architectures prevents unnecessary disruption.

Comparing AI Visibility Tools Across Bing and Google Ecosystems

Bing Citation Share Metrics vs Google Search Console Limits

Citation Share reports the percentage of AI citations a site captures for a given grounding query. This metric appears in Bing Webmaster Tools alongside Intents and Topics to group queries, addressing current dashboard data limits. The Compare feature overlays past periods to track trend shifts. These tools provide a view of AI visibility trends rather than simple citation status. Google Search Console offers no citation-style counts, leaving operators without native grounding data. The platform relies on standard indexing signals rather than generative attribution metrics. This asymmetry means visibility data is currently limited to Bing, covering Copilot and Bing's own answers, and says nothing about Google.

Monitoring efforts must account for the fact that Citation Share data covers only Copilot and Bing answers. Because Google Search Console still offers no citation-style counts, practitioners currently lack native grounding data for that system. Without cross-platform synthesis, visibility reports remain fragmented by vendor silo.

Deploying Bing Compare Features for Period-Over-Period AI Analysis

The Compare function allows users to overlay a past period over the present one to isolate trend shifts in AI visibility. This capability addresses the latency inherent in generative indexing, where content changes may not reflect immediately in citation counts. Operators must recognize that Bing Webmaster Tools currently limits this data to Copilot and Bing answers, excluding Google ecosystems where native citation metrics remain unavailable.

Strategic reliance on a single vendor's preview tools limits visibility to Bing data, as coverage spans only Copilot and Bing answers. While the feature rolls out globally, the preview status implies potential data volatility during high-frequency updates. The absence of comparable native tools in other engines necessitates a unified measurement framework rather than siloed dashboard reviews.

Why llms.txt Fails to Influence LLM Differentiation According to Google

Self-reported llms.txt files cannot help an LLM differentiate one site from another because the data lacks independent verification. John Mueller argued that LLM systems cannot use these documents to decide which websites to surface for a given query. Since the site hoping to be chosen authors the file, the mechanism offers no objective signal for ranking one domain over a competitor; he pointed back to ordinary HTML and internal links instead. This stark utilization gap confirms that structured files currently fail to influence AI visibility in production environments. Operators relying on these files for discovery face a structural limitation where the format does not translate to citation frequency. The implication for search teams is clear: maintaining these files remains low-cost, yet expecting them to drive differentiation is unfounded without broader system adoption.

Given that llms.txt cannot make an LLM choose a site and citation-generating bots barely fetch the file, the immediate takeaway is to audit fetch logs rather than assuming file presence equates to agent consumption.

Implementing Compliant AI Discovery Strategies Under New Regulations

UK Fair Ranking Rules and Objective Criteria Definitions

Mandates from the UK Competition and Markets Authority now force Google Search to rank organic results, including AI Overviews, using objective criteria while explicitly excluding ads from these constraints. This Fair Ranking requirement forces a shift from opaque algorithmic adjustments to verifiable, non-discriminatory logic for result ordering. Operators must now expect advance notice of significant changes, effectively ending the era of unannounced core updates impacting visibility without recourse. The route to raise ranking concerns is now obligatory, creating a feedback loop previously absent in search operations. Google disputes the necessity by claiming existing transparency exists, yet the rules apply strictly within the UK jurisdiction following an early-June opt-out deadline. The distinction between organic results and paid placements remains the critical boundary for compliance enforcement. This framework compels a move toward documented ranking factors over assumed heuristics. The operational burden shifts to proving neutrality rather than merely asserting it. Search teams must audit their visibility metrics against these new disclosure windows. The definition of fairness now includes procedural transparency, not outcome equity. Operators should document baseline performance to measure against future Fair Ranking disclosures.

Steps for Publishing OKF Files and llms.txt Setup

Placing a single text file at your domain root defines the deployment of llms.txt, yet this action yields negligible visibility gains for general search. Consequently, operators should treat this format as a low-cost optional artifact for coding agents rather than a primary lever for AI search visibility. The practical utility remains limited to specific training crawlers, meaning broad deployment strategies should prioritize content quality over file presence. Publishing OKF files follows a similar static path but targets a different agent class focused on organizational knowledge and runbooks. Since the Open Knowledge Format is currently at version 0.1, adoption involves manual file creation as the specification remains in an early draft stage.

Validation Checklist for llms.txt Deployment and CMA Compliance

Verify llms.txt presence for specific agent use cases while prioritizing the statutory route to raise ranking concerns mandated by UK regulators. This dual approach addresses technical configuration without conflating file placement with compliance guarantees. The Fair Ranking requirement explicitly covers organic results, including AI Overviews, but excludes ads from these non-discriminatory constraints. Retrieval bots generating citations account for a negligible fraction of fetches, so treat the structured file as a low-value artifact for general visibility. A common misconception involves assuming file deployment satisfies the legal obligation to provide objective criteria for result ordering. UK rules demand a functional mechanism for operators to challenge ranking logic, a procedural safeguard no static text file provides. The file format remains cheap to maintain yet offers no use over how systems differentiate sites for discovery. Organizations relying solely on llms.txt setup miss the substantive shift toward verifiable fairness in search operations. True compliance requires establishing clear channels for dispute resolution rather than optimizing for agents that rarely visit. Experts recommend auditing grievance procedures immediately to meet the new regulatory baseline for organic visibility.

References

• The 9 Best AI Visibility Tools Tested and Ranked
• AI Visibility for B2B SaaS: The 2026 Complete Guide
• LLM-Friendly Content: 12 Tips to Get Cited in AI