Common Crawl analyzes 584K llms.txt files, finds most lack substance and misuse crawler directives
TLDR
Common Crawl's analysis of over 584,000 llms.txt files reveals most are template-generated, many contain no links, and some misapply crawler-control rules the format doesn't support.
Optimixed Overview
Optimixed's original report on this story.
Common Crawl examined 584,107 llms.txt files and found that the vast majority originated from templates rather than deliberate site configuration, with many lacking actual link content and some misusing crawler directives the format cannot enforce.