The internet’s biggest “security guard” is about to become its top cashier. On July 1, Cloudflare published a blog post with a mild title: Your Website, Your Rules. Its contents, however, are anything but lenient. Starting September 15, all websites using Cloudflare will block multi-purpose AI crawlers by default. If your pages host advertisements, AI training crawlers and AI Agent crawlers will be locked out entirely—unless you manually lift the restriction from your backend dashboard.
Notice this total reversal of logic: previously, AI bots were permitted by default, with website owners able to opt for blocking. The new rule flips this on its head: all AI crawlers get blocked automatically, and owners must manually enable access if they wish to allow them.
This marks the first systemic “regulation” of AI data harvesting at the foundational infrastructure layer of the entire internet.
The policy stems from a landmark shift: automated bot traffic has outpaced genuine human traffic online.
Matthew Prince, CEO of Cloudflare, stated this milestone arrived far earlier than projected; the firm originally predicted it would not occur until 2027. In plain terms, most web pages you open today are viewed primarily by machines, not people.
How AI-driven traffic is regulated will shape both the future of every website and Cloudflare’s own business trajectory as the web’s gatekeeper.
1. The Strictest Crawler Policy Ever Created
Per Cloudflare’s official breakdown, AI crawlers fall into three distinct categorized groups:
- Search Crawlers: Traditional indexing bots built for search engines—the type Google has operated for over two decades.
- Agent Crawlers: AI agents that browse the web on users’ behalf in real time. For instance, when you ask ChatGPT to look up facts or complete online forms, an Agent crawler retrieves that data for you.
- Training Crawlers: Bots that scrape massive volumes of web content exclusively to train large language models.
The three bot types are labeled separately, letting site administrators toggle “allow” or “block” permissions for each category independently. You can open access to search engines for discoverability, permit AI Agents to assist your visitors, or fully block Training crawlers to stop AI firms from harvesting your content for model development without compensation.
This three-tier classification directly targets Google’s business model.
Googlebot functions as a hybrid crawler: it indexes pages for Google Search while simultaneously collecting data to power Google’s native AI tools such as AI Overviews. Google does offer a separate Google-Extended tool that lets publishers opt out of AI training data collection, yet the core Googlebot itself continues to siphon content for the search engine’s built-in AI features with no separation.
Google’s architecture never truly decouples search indexing from AI data gathering.
The practical consequence, backed by Cloudflare’s internal data, is clear: websites rely on Google Search visibility to attract audiences, so they cannot block Googlebot. Once Googlebot gains entry, the platform automatically scrapes content for AI training alongside search indexing. This grants Google roughly twice as much access to web content as competing AI companies.
Cloudflare has also instituted a “strictest rule priority” clause. If a single crawler performs both search indexing and model training functions, all applicable access controls activate simultaneously, enforcing the harshest restriction available. In short, if you block Training crawlers, hybrid bots including Googlebot, Applebot and BingBot will all be blocked as a side effect.
This policy dismantles Google’s bundled tradeoff: the requirement to accept AI training data scraping in exchange for search engine exposure. Cloudflare argues this bundling is unfair and must be split apart.
A set of ratios illustrates how the old web content compact has completely collapsed. Cloudflare released crawl-to-referral traffic metrics for major AI platforms: Google sits at approximately 14:1—14 page crawls for every one visitor click sent back to the publisher. OpenAI’s ratio hits 1,700:1, while Anthropic’s stands at an extreme 73,000:1.
The original search engine bargain was simple: search engines scrape your content, and you gain referral traffic. That transaction no longer balances out in the AI era.
2. From Web Security Guard to Content Cashier
If Cloudflare only shielded websites from unauthorized AI crawlers, the policy would amount to nothing more than a defensive tool. But Cloudflare has far grander ambitions beyond acting as a passive security service.
Last July, Cloudflare rolled out its Pay Per Crawl program, charging AI firms fees based on each individual crawl request. This year, the platform upgraded the model to Pay Per Use. The key difference lies in payment triggers: instead of charging per bot visit, publishers only receive compensation when their content delivers tangible value within an AI system—such as generating a chatbot response or appearing within AI search results.
The shift from per-request billing to value-based pricing reveals a massive strategic goal: Cloudflare aims to build an open marketplace, not just a digital barrier wall.
Its initial partner roster includes two AI search startups: Ceramic.ai and You.com. Publishers who opt into the program earn payouts whenever their content surfaces in Ceramic’s AI search results or is accessed by You.com’s AI Agents. Major media groups have voiced public support: Condé Nast’s CEO called the model a “game-changer,” while Reddit’s co-founder stated the entire online ecosystem will benefit.
On the surface, this seems like a flawless solution, yet one critical flaw complicates the narrative.
Back in March, Cloudflare launched its own crawler API. Users submit a URL, and the tool scrapes an entire website in one batch, returning content formatted as HTML, Markdown or structured JSON. This move sparked widespread unease among publishers: the company trusted to block rogue crawlers had built a scraping bot of its own.
Matters grew more awkward when multiple publishers reported their Cloudflare crawler block rules failed to function as intended. Cloudflare later patched the bug, but the industry narrative had already solidified: “We protect websites from scrapers—except our own.”
Cloudflare defends its native crawler as fully compliant, stating it adheres to robots.txt protocols and the platform’s AI Crawl Control rules. If a site admin blocks AI crawlers site-wide, Cloudflare’s proprietary bot will also be denied access. One developer summed up the strategy as a “hedged bet that guarantees Cloudflare wins either way.”
This raises a fundamental conflict of interest: is Cloudflare a neutral, impartial infrastructure referee, or a new breed of middleman?
The evidence leans toward the latter.
It simultaneously occupies three conflicting roles: rulemaker (creating the three-tier crawler classification system), rule enforcer (blocking bots at the network infrastructure level), and market participant (running its own scraping API and content monetization marketplace).
This is not to dismiss the policy’s merits. Moving unregulated, exploitative AI scraping into a structured, permission-based framework with clear bot categorization represents genuine progress. Still, labeling Cloudflare a savior for all content creators is overly naive.
The firm is constructing an AI content toll station with itself positioned as the central hub.
3. Will Independent Creators Get a Slice of the Revenue Pie?
This section covers the most sobering reality of Cloudflare’s new policy.
Condé Nast, Dotdash Meredith, Reddit—every major publisher and social platform publicly endorsing the framework possesses massive content libraries, dedicated legal teams and substantial negotiating leverage. Even without Cloudflare’s tools, these corporations are fully capable of striking direct licensing deals with AI developers; over 50 large-scale content licensing agreements have been finalized globally in the past year. For big media brands, Cloudflare merely adds an optional tool, not an essential lifeline.
What about independent creators? A solo developer posting technical tutorials on WordPress, a self-media creator publishing in-depth analysis on Chinese WeChat Official Accounts?
In theory, Cloudflare’s global infrastructure removes the need for individual creators to negotiate separate contracts with every AI company, streamlining permission controls and revenue collection. The phrase “in theory” is critical here. Pay Per Use currently only partners with Ceramic.ai and You.com, two minor niche platforms. None of the major content-consuming AI leaders—OpenAI, Google, Anthropic—have joined the program.
A deeper practical contradiction exists for small creators: online visibility is their scarcest asset. Blocking AI crawlers risks eliminating all chances of discovery by AI tools. When large media outlets restrict bot access, Google Search still indexes their content. A tiny independent blog that blocks crawlers may vanish entirely amid the internet’s endless noise.
Disturbing traffic data underscores this imbalance:
AI chatbots drive roughly 96% less referral traffic than traditional search engines. Only around 1% of users click embedded source links within AI-generated answers. Publishers have lost between 20% and 90% of their web traffic and advertising revenue over the past year due to AI search tools. Separate research found Google’s AI Overviews cut external website click-through rates by approximately 40%.
This means even if Pay Per Use scales widely, the generated payouts will almost certainly fail to offset the advertising revenue publishers have already forfeited. The system amounts to damage limitation at best, and not guaranteed damage limitation at that.
Cloudflare’s internal analysis notes over half of all AI crawler bandwidth is wasted re-scraping web pages that have not been updated. Streamlining this inefficient traffic does deliver tangible value, yet optimizing crawl efficiency and letting creators earn sustainable income are two entirely separate challenges.
4. Even Cyber Bodhisattvas Have Their Own Temples
Cloudflare earned its affectionate moniker “Cyber Bodhisattva” for one key contribution: it drags opaque, unregulated AI data exploitation into the open, forcing AI developers to disclose exactly how they intend to use scraped web content. At a time when automated bot traffic dominates the internet, stepping forward to demand standardized ground rules deserves recognition.
That said, every Bodhisattva maintains its own temple and interests.
Cloudflare manages approximately 20% of all global web traffic—a sizable share, yet far from universal. The remaining 80% of websites operate outside its protective network. AI companies can simply shift their large-scale data harvesting operations toward non-Cloudflare-hosted sites to bypass restrictions.
Google and Apple already provide nominal opt-out tools for AI training data collection, creating potential loopholes to evade Cloudflare’s bot blocks. The UK’s Competition and Markets Authority (CMA) is also pressuring Google via regulatory channels to implement a true opt-out system that does not harm publishers’ search ranking positions.
A single infrastructure provider’s policy cannot finalize this massive redistribution of digital content ownership rights.
Still, the shift reveals a far larger industry trend: the internet’s primary tollbooths are migrating from search engine platforms down to the core network infrastructure layer.
For the past two decades, Google stood as the central gatekeeper dictating which web content gains visibility. Now Cloudflare aims to install a checkpoint at a deeper network level: any automated crawler accessing your site must declare its purpose upfront and comply with established access rules.
The location of the toll station has changed—but the entity collecting the toll remains a powerful commercial intermediary.
© 版权声明
All copyrights of this article belong to the author. No reproduction without permission。
THE END
