Optimixed Search News

Cloudflare adds AI crawler management tool that lets sites set policy by category instead of individual bot

55
News

TLDR

Cloudflare launched Bot Preference Sync, allowing sites to set AI crawler access policies by category rather than managing individual crawlers in robots.txt.

Optimixed Overview

Optimixed's original report on this story.

Cloudflare introduced Bot Preference Sync, a tool that lets sites manage AI crawler access by category type—such as allowing GPTBot while blocking Bytespider—rather than writing individual rules for each crawler in robots.txt. The feature simplifies crawler policy management by grouping bots into categories instead of requiring site owners to maintain separate rules for every individual AI crawler.

Topics:AI & Machine Learning in SEO
Was this useful?
Share

Related articles

65
News

Cloudflare Tool Lets Sites Block AI Training While Keeping Google Access

Cloudflare introduced a Disallow AI Training option that blocks AI crawlers from training data while preserving access for Googlebot and other search engines.

Technical SEOSEO News & Algorithm UpdatesAI & Machine Learning in SEO
55
News

Google Tests Ad Metrics in Chrome User Experience Report

Google is testing new ad metrics—Ad Count, Ad Density, Ad Weight (Network), and Ad Weight (CPU)—in CrUX to measure page ad impact.

Technical SEOSEO News & Algorithm UpdatesAnalytics & Measurement
65
News

Google on Daily-Crawl Sitemap Hack: Why It Backfires

Google's John Mueller addressed an SEO technique using sitemaps to force daily crawling, explaining why the approach doesn't work as intended.

Technical SEOSEO News & Algorithm Updates
55
News

How AI Cyberattacks Are Reshaping Website Security for SEO

Over 100 organizations including OpenAI, Google, and Microsoft are warning website owners to prepare for escalating AI-driven cyberattacks.

Technical SEOSEO News & Algorithm UpdatesAI & Machine Learning in SEO