The foundational deal of the modern web, a handshake agreement that powered two decades of search and content, is officially dead. Cloudflare just put a price on scraping the internet, and it’s coming for artificial intelligence’s free lunch.
Almost 30 years ago, two Stanford grad students, Larry Page and Sergey Brin, built Google on a simple bargain: content creators would let them copy the entire web in exchange for traffic. For years, that traffic powered ad revenue, subscriptions, and the growth of online media. Google mostly upheld its end of the deal. But that era is collapsing under the weight of AI.
On July 1, Cloudflare, one of the internet’s core infrastructure companies, declared “Content Independence Day.” In a landmark policy shift, the company announced it will now block AI crawlers from scraping sites hosted on its platform unless those bots pay content creators for the data they consume.
“Cloudflare, along with a majority of the world’s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content,” CEO Matthew Prince announced in a blog post. “That content is the fuel that powers AI engines, and so it’s only fair that content creators are compensated directly for it.”
This is a sharp, aggressive turn from the web’s traditionally open access ethos. Cloudflare argues it’s long overdue. AI tools like ChatGPT and Google’s own AI Overviews are now answering user questions directly, effectively strip-mining websites for information while sending almost no traffic back to the original source.
“Instead of being a fair trade, the web is being stripmined by AI crawlers with content creators seeing almost no traffic and therefore almost no value,” Prince said.
Publishers, we see you! 🙌 Cloudflare just launched pay per crawl to put control over your content back where it belongs.
Now, crawling is more transparent and controlled, by default, creating a better web ecosystem for creators like you. This is about real content… pic.twitter.com/yatB5LSBIm
— Cloudflare (@Cloudflare) July 2, 2025
The Web as Swiss Cheese
The numbers are stark. Cloudflare claims it’s already 10 times harder to get traffic from Google than it was a decade ago due to features like the answer box. But the new AI models are far worse. According to Cloudflare’s internal metrics, OpenAI drives 750 times less traffic than traditional Google search, while Anthropic drives a staggering 30,000 times less. The reason is simple: people are asking ChatGPT instead of Googling. The content still gets used, but the creators have been completely cut out of the value chain.
Using its position as a gatekeeper for roughly 20% of all websites globally (around one-fifth of all web traffic passes through Cloudflare’s network), Cloudflare is now forcing the issue by charging a toll.
But the plan goes further than just blocking bots. Cloudflare aims to build a new content marketplace where AI companies and creators can trade directly. Compensation would be based not on clicks, but on how valuable the content is for training AI models. To explain this, the company uses a quirky metaphor: imagine an AI’s knowledge is a block of Swiss cheese. The holes represent knowledge gaps. The more your content fills one of those holes, the more it’s worth to an AI company.
It’s an ambitious proposal that challenges the entire web economy, which still judges value by how viral something is. Cloudflare is betting that filling gaps in machine knowledge is a more stable long-term market than chasing fickle human attention. It also hints at something more radical: the end of the free and open web as we knew it.
This move marks the dawn of the pay-to-train era. OpenAI has already signed high-profile licensing deals with publishers like Reddit and the Financial Times. Other AI giants are quietly inking data partnerships or scraping whatever they can until they get blocked. But Cloudflare’s decision is the first time a major infrastructure provider has flipped the default setting for a huge portion of the internet.
Our Take
The real story here is not just technical; it is economical. We are watching the rise of a new class of digital middlemen, companies that will broker access between the creators of web content and the AI models that feed on it. In a post-click internet, training data is the new currency, and Cloudflare just positioned itself as a major bank.
The company says its goal is to usher in a new golden age for creators. “We believe that if we can begin to score and value content not on how much traffic it generates, but on how much it furthers knowledge,” Prince said, “we not only will help AI engines get better faster, but also potentially facilitate a new golden age of high-value content creation.”
That sounds nice. But it also raises messy questions. Who decides what counts as high value? Who gets paid, and how much? If content is optimized for AI rather than people, what happens to the soul of the web? The darker possibility is a content Cold War, where publishers wall off everything and AI companies hoard exclusive data deals, making the web more fragmented and less open than ever before.
Whether or not Cloudflare’s “Swiss cheese” model takes off, this much is true: AI broke the old search-based web economy. On July 1, Cloudflare drew a line in the sand. For the first time in the age of generative AI, the pipes of the internet are fighting back.
Read the full article here
