Optimixed Search News

Common Crawl analyzes 584K llms.txt files, finds most lack substance and misuse crawler directives

By the Optimixed Newsroom
52
News

TLDR

Common Crawl's analysis of over 584,000 llms.txt files reveals most are template-generated, many contain no links, and some misapply crawler-control rules the format doesn't support.

Optimixed Overview

Optimixed's original report on this story.

Common Crawl examined 584,107 llms.txt files and found that the vast majority originated from templates rather than deliberate site configuration, with many lacking actual link content and some misusing crawler directives the format cannot enforce.

Topics:AI & Machine Learning in SEO
Was this useful?
Share

Related articles

35
News

OpenAI Explains ChatGPT's LLM-Based Ad Selection System

OpenAI disclosed how ChatGPT's LLM-powered ad system interprets conversations and selects ads for auction.

AI & Machine Learning in SEOPaid Search & AdvertisingTools & Platforms
65
News

Google CEO: AI Agents Will Browse Sites via Pages or Protocols

Google CEO Sundar Pichai outlined two paths for AI agents to access websites: direct page browsing and protocol-based methods, with protocols offering better scalability but longer development cycles.

AI & Machine Learning in SEOSEO News & Algorithm Updates
65
Analysis

ChatGPT Drives Under 1% of Google Organic Search Traffic

Analysis of 100 websites' GA4 data shows ChatGPT referrals accounted for just 0.62% of Google organic search traffic over 12 months, with referral growth stalling after August 2025.

AI & Machine Learning in SEOAnalytics & MeasurementContent & Strategy
28
News

Walmart Launches AI Agents for Autonomous Shopping

Walmart is rolling out AI agents that can autonomously make purchases on customers' behalf within pre-set spending limits.

AI & Machine Learning in SEOEcommerce MarketingTools & Platforms