Optimixed Search News

Common Crawl analyzes 584K llms.txt files, finds most lack substance and misuse crawler directives

52
News

TLDR

Common Crawl's analysis of over 584,000 llms.txt files reveals most are template-generated, many contain no links, and some misapply crawler-control rules the format doesn't support.

Optimixed Overview

Optimixed's original report on this story.

Common Crawl examined 584,107 llms.txt files and found that the vast majority originated from templates rather than deliberate site configuration, with many lacking actual link content and some misusing crawler directives the format cannot enforce.

Was this useful?
Share

Related articles

52
Research

Semrush found that manufacturing brands cited in AI search results don't see corresponding traffic gains from those mentions.

AI & Machine Learning in SEOSEO News & Algorithm UpdatesAnalytics & Measurement
25
Interview

Lenny's Newsletter interviews Roman Ugarte, who led the rapid development of SpaceX AI's Grok Bot knowledge-work agent, discussing the product decisions and team structure that drove its launch.

AI & Machine Learning in SEOTools & PlatformsCareer & Industry Trends
35
Analysis

MARTECH reports that while AI accelerates early-stage content drafting, marketing teams still lose time to approval workflows, team coordination, and decision overload—suggesting the real bottleneck was never copywriting.

AI & Machine Learning in SEOContent & StrategyTools & Platforms
52
News

Google is experimenting with paginated follow-up questions and responses within AI Mode, allowing users to skip them.

AI & Machine Learning in SEOSEO News & Algorithm Updates