Technology · Big Tech AI investment, earnings and data centre expansion
OpenAI, Google, Microsoft, Anthropic pledge shared opt-out signal for AI training scraping
OpenAI, Google, Microsoft and Anthropic said on 30 October they will adopt a single robots.txt-based signal letting publishers opt out of AI training. Anthropic has started; Microsoft, Google and OpenAI will follow in 2027 or the coming months.
Updated (version 2). New coverage since the last version from CNA.
1 / 3
The story, neutrally told
Mixed · 2OpenAI, Google, Microsoft and Anthropic said on 30 October that they will adopt a common standard that lets publishers opt out of having their content scraped to train AI models. The VergeLC “OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI” Read at The Verge ↗ The Straits TimesRC “OpenAI, Google, Microsoft and Anthropic pledged on Oct 30 to give media outlets and other websites a single way to prevent their content from being scraped” Read at The Straits Times ↗ Mixed · 2At present, publishers must signal their AI training preferences separately to each company; the new signal would let them set one preference that every adopting company honours. The VergeLC “Currently, sites have to signal their AI training preferences separately to each AI company.” Read at The Verge ↗ The Straits TimesRC “a single, standardised instruction that is recognised and honoured by any AI company that adopts the protocol” Read at The Straits Times ↗ Left · 1The Verge reports that the "policy token" signal was proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF), and will sit within a site's robots.txt file. The VergeLC “The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF).” Read at The Verge ↗
Mixed · 2Rollout is staggered. Anthropic says it has already started; Microsoft will implement it from 1 February 2027; Google will do so sometime in 2027; and OpenAI says "in the coming months". The Straits TimesRC “Microsoft will implement it beginning Feb 1, 2027, and Google sometime in 2027.” Read at The Straits Times ↗ The VergeLC “While Anthropic has already adopted the signal, Google and Microsoft plan to support it next year. OpenAI says it will implement it in the “coming months.”” Read at The Verge ↗ Mixed · 2According to AFP copy carried by The Straits Times and CNA, Microsoft said in a blog post that the setting applies only to future model training, does not stop bots from crawling otherwise, and leaves scraping allowed by default until a publisher changes it. CNAN “The new setting will apply only to future training of models but does not prevent bots from crawling otherwise, and content scraping will be allowed by default until the setting is changed by a publisher, Microsoft said in a blog post.” Read at CNA ↗ The Straits TimesRC “content scraping will be allowed by default until the setting is changed by a publisher, Microsoft said in a blog post.” Read at The Straits Times ↗ Left · 1Microsoft will offer finer controls, letting publishers opt out of AI training or of AI grounding (pointing a model to a verified source to reduce hallucinations), and says sites will still appear in Bing search results regardless. The VergeLC “Microsoft, however, will offer more granular controls that will give publishers the ability to opt out of AI training or AI grounding”“Microsoft notes that websites will still appear in Bing’s search results regardless of their preferences.” Read at The Verge ↗
Right · 1Microsoft and Google warned in separate blog posts that opting out of AI training could reduce the chances of a publisher's content surfacing in AI apps and experiences. The Straits TimesRC “Opting out of AI training could reduce the chances that a publisher’s content will surface in AI apps and experiences, Microsoft and Google said in separate blog posts.” Read at The Straits Times ↗ Mixed · 2The move comes amid a surge in AI crawlers and legal fights: the New York Times sued OpenAI and Microsoft in December 2023, and the companies cite fair use. Cloudflare has blocked AI bots by default on new customer sites since July 2025, and Really Simple Licensing lets publishers charge crawlers. The Straits TimesRC “In December 2023, the Times sued OpenAI and Microsoft for allegedly using millions of its articles without authorisation or payment to train ChatGPT.” Read at The Straits Times ↗ The VergeLC “the Really Simple Licensing (RSL) standard giving publishers the ability to charge crawlers that visit their sites, and Cloudflare blocking AI crawlers by default” Read at The Verge ↗ Right · 1Meta spokesman Andy Stone wrote on X that a common signal is "an important step" and said Meta looks forward to working with other AI companies on it. The Straits TimesRC “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step.” Read at The Straits Times ↗
Mixed · 2CNA's version of the same report dates the pledge to Friday and gives the date as "Oct 31", while The Straits Times gives Oct 30. CNAN “pledged on Friday (Oct 31) to give media outlets and other websites a single way” Read at CNA ↗ The Straits TimesRC “pledged on Oct 30 to give media outlets and other websites” Read at The Straits Times ↗
Every sentence links to the reporting it rests on. The pill in front of each says where its sources sit: Left, Centre or Right when one side supplies at least half of them, Mixed when they are evenly split. The number is how many outlets it cites.
Left1 outlet
- Framing
- Tech-explainer framing: a positive step towards a single control for publishers, with technical detail on the IETF proposal and robots.txt.
- Emphasis
- Standard's origin (Anthropic's Kevin Kelley, IETF), robots.txt, Microsoft's granular training/grounding controls.
- Leaves out or plays down
- Does not mention the New York Times lawsuit, the default-allow setting, or the warning that opting out may reduce visibility in AI products.
- For example
-
“The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF).” — The Verge
Centre1 outlet
- Framing
- CNA runs the same report as The Straits Times, framed as a pledge with limits: a single instruction for publishers, but slow to take effect and covering only some scraping.
- Emphasis
- Single standardised instruction, months-long rollout, default-allow, the New York Times suit and fair-use defence, and the visibility trade-off.
- Leaves out or plays down
- Does not name the IETF or Kevin Kelley, or Microsoft's separate training and grounding options.
- Charged language
- “block some scraping”
Right1 outlet
- Framing
- AFP wire copy carried by The Straits Times, framed as a pledge with limits: it will take months, covers only some scraping, and follows publisher litigation.
- Emphasis
- Delayed timelines, default-allow, no block on crawling, the New York Times suit and fair-use defence, visibility trade-off.
- Leaves out or plays down
- Does not name the IETF or Kevin Kelley, or Microsoft's grounding option.
- For example
-
“The solution will not fully take effect for months, though.” — The Straits Times
What every side reports
- OpenAI, Google, Microsoft and Anthropic announced they will adopt a shared opt-out signal for AI training.
- Publishers currently must set preferences separately for each AI company.
- Anthropic has begun adoption; Microsoft and Google follow in 2027; OpenAI says 'in the coming months'.
- Cloudflare's default AI-bot blocking and Really Simple Licensing are existing measures.
Where accounts differ
-
What the signal does to crawlers
- Left
- The Verge says the four companies will restrict their AI crawlers from accessing a site that carries the token.
- Right
- AFP says the setting applies only to future training and does not prevent bots from crawling otherwise; scraping stays allowed by default.
-
OpenAI, Google, Microsoft and Anthropic will adopt a single shared signal letting publishers opt out of AI training across companies.
Reported- Reports 3
- The Verge, CNA, The Straits Times
-
The shared opt-out signal is being rolled out in stages: Anthropic has started, OpenAI is due in the coming months, Microsoft from 1 February 2027 and Google sometime in 2027.
Reported- Reports 3
- The Verge, CNA, The Straits Times
Anthropic organisation
Has already started rolling out the signal; one of its standards leads proposed it.
“Claude-maker Anthropic said it has “already started” rolling it out.” — The Straits Times
Microsoft organisation
Will implement from 1 Feb 2027 with finer controls; says scraping stays allowed by default and opting out may reduce visibility in AI experiences.
“Microsoft will implement it beginning Feb 1, 2027” — The Straits Times
Google organisation
Will support it in 2027 and warns that opting out may reduce content's appearance in AI experiences.
“Opting out of AI training could reduce the chances that a publisher’s content will surface in AI apps and experiences, Microsoft and Google said in separate blog posts.” — The Straits Times
OpenAI organisation
Says it will implement the signal in the coming months; is being sued by the New York Times and cites fair use.
“ChatGPT-maker OpenAI said it will roll it out “in the coming months”” — The Straits Times
Left1 article
-
The biggest names in AI are backing a new way to opt out of scraping
Neutral Concise explainer on the shared policy token and its technical home in robots.txt.
Centre1 article
-
CNAN ·
AI giants promise publishers new way to block some scraping
Neutral Text matching the AFP report run by The Straits Times, presenting the pledge as real but delayed and limited, set against publisher lawsuits.
Right1 article
-
The Straits TimesRC · · via AFP
AI giants promise publishers new way to block some scraping
Mixed AFP wire presenting the pledge alongside its delays, limits and publisher litigation.
- 30 Oct 20:26 First The VergeLC The biggest names in AI are backing a new way to opt out of scraping
- 31 Oct 01:35 +5h 9m The Straits TimesRC AI giants promise publishers new way to block some scraping via AFP
- 31 Oct 05:47 +9h 21m CNAN AI giants promise publishers new way to block some scraping
Times are when each article was published, or when we first saw it if the outlet gave no time.
Ask about this story
Loading…