Back to AI News
AIHUB

Google and Microsoft join OpenAI's new website crawling standard for opting out of AI training

OpenAI has proposed a new website crawling standard that allows sites to opt out of AI training via a single robots.txt line. Google, Microsoft, and Anthropic are among the companies signing up to the plan. The standard only prevents training, not crawling, so bots may still visit sites for other purposes.

AI Brief

This is a concrete industry development with major players aligning on a new technical standard that affects AI data sourcing and web governance.

One robots.txt line could soon tell Google, Microsoft, OpenAI, and Anthropic not to train on your site, but it won't stop their bots from crawling it.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!