OpenAI Clarifies Its Web Crawlers Can Update User-Agent Strings
Optimixed.com
SERoundtable
32
News
TLDR
OpenAI updated its crawler documentation to clarify that its user-agent string version number may change and that it uses an additional robots.txt marker when fetching robots.txt files.
Optimixed Overview
Optimixed's original report on this story.
SERoundtable reports that OpenAI made clarification changes to its crawler help documentation, noting that the version number of the user-agent string may be updated and that OpenAI may use a user-agent string with an additional robots.txt marker to help site owners distinguish requests for robots.txt files in their logs. Barry Schwartz notes that while the updates themselves are not particularly novel, the documentation changes were worth highlighting to the SEO community.
Was this useful?