Publishers are facing very tough choices when it comes to Google’s AI, according to a new Wall Street Journal report. Facing a drought of search traffic, they’re considering cutting off Google’s access to their content in various ways. Some even seem to be considering the drastic step of completely cutting off Google’s web crawlers completely.
Anonymous sources told the Journal that Reddit “has discussed shutting off the technology giant’s access to its content for AI use,” to use the Journal’s phrasing. This would represent a big change of heart, since Google pays Reddit $60 million per year to scrape its content for AI, per a deal Reddit signed in 2024.
But it’s a fairly sweeping report, and the Journal found that a large number of publishers are looking into cutting ties with Google in more serious ways, and many are pretty vocal about it:
- The President of Reuters said, “We are certainly looking at the economic trade-offs between search and AI summaries.”
- Mike Reed, CEO of USA Today said “It’s time to take a stand and say enough is enough.”
- Neil Vogel, CEO of People, Inc. said “Turning them off and blocking them entirely is 100% on the table.”
- Politico and the Economist are also weighing their options when it comes to Google’s web crawlers.
The journal puts the dilemma in these terms:
“Publishers can turn off crawling specifically for training Google’s AI models, but they generally must allow the bot to crawl for both AI query results and the traditional search function if they want their content featured in either.”
Drilling down a bit, according to BotDetector, this may not be totally necessary if the goal is to stay out of Google Overviews, and to not have your content used as AI training data. Blocking the Google-Extended feature of the bot is supposed to stop content from being used to train models like those in the Gemini family. To theoretically be blocked from Google’s AI Overviews, BotDetector tells me you have to get a little more intense and use the “nosnippet” tag, which would have the potential to hurt search.
But even that would be far less dramatic than, say, using the much more nuclear “noindex” tag. That one is used to block search indexing altogether. If a critical mass of publishers were seriously considering using that, it would be a major escalation.
A spokesperson for Google told the Journal its AI sends “billions of clicks to the web every week,” and added that they “highlight links to the web and help creators and publishers grow their audiences, and we offer clear controls for website owners to manage their content.”
Read the full article here
