Optimixed Search News

OpenAI Claims Robots.txt May Not Stop ChatGPT's Web-Crawling Bot

72
News

TLDR

SEOFOMO News reports that OpenAI's documentation suggests ChatGPT's Fetch Bot may not be bound by robots.txt restrictions, despite evidence of it accessing blocked sites.

Optimixed Overview

Optimixed's original report on this story.

OpenAI's own documentation indicates that ChatGPT's Fetch Bot may not honor robots.txt directives that would normally prevent crawler access. Per SEOFOMO News, new data shows the bot is reaching sites that have explicitly disallowed it, raising questions about whether traditional crawl-blocking methods remain effective against AI training infrastructure.

Topics:AI & Machine Learning in SEOSEO News & Algorithm Updates
Was this useful?
Share

Related articles

35
Analysis

Local AI Compute for SEO: Chrome Extension Experiment

A Chrome extension experiment demonstrates that local AI models like Gemini Nano can handle certain SEO workflows without requiring cloud-based frontier models.

AI & Machine Learning in SEOTools & PlatformsTechnical SEO
65
News

Google AI Overviews Now Appearing for Major Brand Searches

SERoundtable observes that Google is showing AI Overviews more frequently for searches on major brand names, with exceptions for Google's own brand and news publication queries.

AI & Machine Learning in SEOSEO News & Algorithm Updates
35
Analysis

OpenAI's 2026 Dev Day: New models, APIs, and tools for developers

Lenny's Newsletter covered OpenAI's Dev Day 2026, where the company unveiled new tools including Dots, Spaces, Sites, GPT-6.1 Sol, an updated Decisions API with vision capabilities, and Astra ultrafast.

AI & Machine Learning in SEOTools & PlatformsContent & Strategy
35
News

OpenAI Dots Now Run Read-Only Research Checks

OpenAI has enabled its Dots product to automatically run read-only research checks on connected apps between user conversations.

AI & Machine Learning in SEOTools & Platforms