Optimixed Search News

OpenAI Clarifies Its Web Crawlers Can Update User-Agent Strings

32
News

TLDR

OpenAI updated its crawler documentation to clarify that its user-agent string version number may change and that it uses an additional robots.txt marker when fetching robots.txt files.

Optimixed Overview

Optimixed's original report on this story.

SERoundtable reports that OpenAI made clarification changes to its crawler help documentation, noting that the version number of the user-agent string may be updated and that OpenAI may use a user-agent string with an additional robots.txt marker to help site owners distinguish requests for robots.txt files in their logs. Barry Schwartz notes that while the updates themselves are not particularly novel, the documentation changes were worth highlighting to the SEO community.

Topics:SEO News & Algorithm UpdatesAI & Machine Learning in SEO
Was this useful?
Share

Related articles

62
Interview

Google SVP on DMA Changes: "Terrible" Hit to Search Quality

Nick Fox, Google's SVP of Knowledge & Information, told an interviewer that EU regulations forced Google to reduce search quality in ways that made him feel "terrible" and "sad."

SEO News & Algorithm UpdatesAI & Machine Learning in SEOContent & Strategy
35
News

Google Tests Two-Row Button Layout in Mobile Local Panels

Google is experimenting with a two-row button layout for local search panel actions on mobile devices.

SEO News & Algorithm UpdatesLocal SEOTechnical SEO
35
News

Google Maps Bug Misplaces Business Pins in Ocean

Google Maps is placing some service area business pins in the ocean instead of at correct locations.

SEO News & Algorithm UpdatesLocal SEOTools & Platforms
68
News

Google Appeals EU's DMA Order to Share Search History Data

Google is challenging an EU Digital Markets Act ruling that would require it to share anonymized search history data with competitors.

SEO News & Algorithm UpdatesAI & Machine Learning in SEOTools & Platforms