Optimixed Search News

OpenAI Claims Robots.txt May Not Stop ChatGPT's Web-Crawling Bot

72
News

TLDR

SEOFOMO News reports that OpenAI's documentation suggests ChatGPT's Fetch Bot may not be bound by robots.txt restrictions, despite evidence of it accessing blocked sites.

Optimixed Overview

Optimixed's original report on this story.

OpenAI's own documentation indicates that ChatGPT's Fetch Bot may not honor robots.txt directives that would normally prevent crawler access. Per SEOFOMO News, new data shows the bot is reaching sites that have explicitly disallowed it, raising questions about whether traditional crawl-blocking methods remain effective against AI training infrastructure.

Topics:AI & Machine Learning in SEOSEO News & Algorithm Updates
Was this useful?
Share

Related articles

35
Product announcement

Semrush Launches AI Visibility Conference Spotlight in London

Semrush is hosting Spotlight, an AI visibility conference in London on October 13, based on proprietary research claiming AI search converts 4.4x higher than organic and 85% of topics lack clear ownership.

AI & Machine Learning in SEOContent & StrategyTools & Platforms
55
Research

Users Aren't Abandoning Google—They're Using AI Tools Alongside It

New research indicates users are blending traditional Google searches with AI tools rather than entirely switching away from Google.

AI & Machine Learning in SEOSEO News & Algorithm UpdatesAnalytics & Measurement
62
Research

Product Pages Lead AI Citations in B2B Buyers, Study Finds

Ten Speed's research shows product pages receive 24% of AI citations for B2B buyers, significantly outperforming Reddit and YouTube.

AI & Machine Learning in SEOSEO News & Algorithm UpdatesEcommerce Marketing
62
News

Gemini 3.8 Flash in Google AI Mode Cites Sources Less Often

Early users report that Google's new Gemini 3.8 Flash model in AI Mode cites and links to sources less frequently than previous versions.

AI & Machine Learning in SEOSEO News & Algorithm Updates