Cloudflare flips the default — AI crawlers now get blocked unless they pay

Information Technology 4 min 80 sources

The company that sits in front of a fifth of the web is changing the internet's unwritten deal: scrape freely, or pay the publisher. Plus a privacy feature that leaked real emails, an $800M neocloud raise, and employers un-doing their AI layoffs.

Key takeaways

Cloudflare, which sits in front of much of the web, will block AI training crawlers by default from September 15 unless they pay — quietly ending the free-scrape deal AI firms relied on.
An Apple "Hide My Email" bug can reveal the real address the feature is meant to hide, and Apple left it unfixed for over a year.
Money keeps pouring into AI infrastructure — Together AI hit an $8.3B valuation — while some employers are already rehiring the workers they replaced with AI.

Cloudflare changes the web’s oldest bargain

For thirty years the deal was simple: a search engine could crawl your website for free, and in return it sent you readers. On Tuesday, Cloudflare — the company whose servers sit in front of a large share of the world’s websites, filtering traffic and blocking attacks — moved to break that deal for AI [54].

Starting September 15, 2026, Cloudflare will block “mixed-use” AI crawlers by default on pages that carry ads, for all its free customers and any new sites [54]. A mixed-use crawler is one bot doing several jobs at once — fetching a page to answer a search query, and also grabbing it to train a model or feed an AI agent. Cloudflare wants those jobs separated, so a site owner can say yes to one and no to another [54].

The reason is a shift in who’s actually reading the internet. Cloudflare says bots recently passed humans as the majority of web traffic for the first time [54]. And more than half of the crawl traffic from AI bots is re-fetching pages that haven’t changed — pure overhead, no new value [54]. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster,” said co-founder and CEO Matthew Prince [54].

The mechanism is a marketplace Cloudflare calls Pay Per Crawl: publishers can charge AI bots a fee to scrape their pages [54]. This week it added a twist — Pay Per Use — where a publisher gets paid not just when a bot fetches a page, but when that content actually creates value, like appearing in an AI search result [54]. Early partners include the AI search startups Ceramic.ai and You.com [54].

Why it matters for anyone who runs a site or builds on scraped data: the free-scrape default that AI companies quietly relied on is being switched off at the gate, for a big slice of the web at once. If your product depends on crawling the open web, your data-access math changes on September 15. Cloudflare notes Google already has roughly twice the crawl access of other AI firms [54] — so the new tollbooth doesn’t hit everyone equally.

A privacy feature that leaked the thing it hid

Apple’s “Hide My Email” lets you generate a random forwarding address so a website never sees your real one. A security researcher found a bug that can reveal the real address behind the mask — the exact opposite of the feature’s job [21][23]. 404 Media independently confirmed the exploit worked on Monday, June 30 [23].

The uncomfortable part is the timeline: the researcher reported it, and Apple left it unfixed for more than a year [23]. 404 Media is withholding the technical details because the flaw still works [23]. “Hide My Email users deserve to know that it may be possible for attackers to discover their hidden email addresses,” the researcher said [23].

The angle: a privacy tool is only as good as the promise behind it. If you use Hide My Email as a hard wall — separate identities that must never link — treat that wall as soft until Apple ships a fix.

The money keeps flowing to the AI plumbing

Together AI, a company that rents out clusters of Nvidia’s AI chips so others don’t have to buy their own, raised $800 million and reached an $8.3 billion valuation [2]. That’s more than double the $3.3 billion it was worth in early 2025 [2]. Its annual bookings — signed customer commitments — passed $1.15 billion last quarter, with customers including the coding-assistant makers Cursor and Cognition [2].

The round was led by Aramco Ventures, the Saudi oil giant’s investment arm, with Nvidia itself joining in [2]. Behind it sits a real trend: usage of open-source AI models — ones anyone can download and run, versus paying to rent a closed model like GPT — has tripled industry-wide in the past year as buyers hunt for cheaper options [2]. The elsewhere money was even bigger: the UAE’s MGX closed one of the largest AI funds ever at $49 billion [12].

Employers are quietly reversing their AI layoffs

Some companies that cut staff and blamed AI are now rehiring, according to reporting Tuesday [60]. The pattern: the tool did part of a job well in a demo, the humans got let go, and the gap between “handles a task” and “holds a role” showed up in the work that followed [60]. It’s an early, still-anecdotal signal — but a useful counterweight to the layoff announcements that read AI capability as a finished replacement rather than a partial one.

The line worth watching: who governs the models

Two threads pulled the same direction Tuesday. Anthropic’s newest models got a broader global release after a round of US government safety testing [48], and the FT reported Washington is in talks with AI companies over voluntary model standards [1]. Meanwhile a UN panel warned that unchecked AI progress could carry catastrophic risks [8]. The through-line: the rules for powerful AI are being drafted right now, mostly as handshakes rather than law — which means they can bend to whoever’s at the table.

02 · Lesson · why it matters

The default nobody voted for

The web ran for decades on a rule everyone obeyed and no one agreed to — and the moment someone with their hand on the switch flips it, we all learn how much we were standing on it.

A rule that felt like weather

For thirty years, one arrangement held the web together: a bot could copy your page for free, and you accepted it because the bot sent you readers. Nobody signed this. There was no vote, no contract, no law. It was a default — the setting things came in, the way the water ran downhill.

Defaults feel like weather. They seem to be facts about the world rather than choices someone made. And because they feel like facts, we build on top of them without noticing we’re building on top of anything. Whole companies grew up assuming free access to the open web the way you assume the ground stays put.

Then, this week, a company that sits in front of a large slice of the world’s websites announced it would change the setting. Starting in September, AI crawlers get blocked unless they pay. The weather didn’t change. Someone changed a knob.

The knob was always someone’s

Here is the part that’s easy to miss. The free-crawl deal wasn’t a law of nature that just now broke. It was always a choice — it just happened to be a choice that suited everyone at once, so nobody had to make it out loud.

When a choice suits everyone, it disguises itself as a fact. You stop seeing the hand on the knob. You forget there is a knob. And the people who benefit most from the arrangement have the least reason to remind you it could be otherwise.

The AI companies that scraped the web for free weren’t stealing — they were doing the normal, permitted thing. But “normal and permitted” is not the same as “guaranteed.” They were standing on a default, and the default belonged to someone else. The moment that someone decided the arrangement no longer suited them, the ground moved — not because anyone broke the rules, but because the rules were never the scrapers’ to keep.

Who else is standing on it

Widen the frame and it isn’t just the AI giants who were leaning on this default. Every small site that got its traffic from search was leaning on the other half of the same deal — copy me, but send me readers. Startups whose entire product is “we read the open web and tell you what’s on it” were leaning on it hardest of all.

And so are you. When you ask an AI a question and it answers with something it read on a news site, a forum, a small blogger’s post — you are standing on the free-crawl default too. The answer arrives clean and free, and the arrangement that made it free is invisible to you. You are not above this system watching it change. You are a node inside it, and the switch that flipped in September changes what your tools can see and what they’ll cost.

The default is a form of power

Notice where the power actually sits. Not with the company with the most money, or the cleverest engineers. It sits with whoever holds the switch — whoever gets to decide what “normal” is for everyone downstream.

This is the quiet shape under the loud news. The story reads as “Cloudflare versus AI companies.” The deeper thing is: the terms of the whole arrangement were set by a party most people never think about, and that party can reset them. Control of the default is control of the field. The one who decides what the setting is by default rarely has to argue, because most people never change a setting they were handed. They just live inside it and call it the world.

An arrangement can serve the one who sets it and still help the people underneath — publishers may get paid now, which many wanted. Both things are true. The point isn’t that the switch was flipped for good or ill. The point is that it was theirs to flip, and yours to live with.

What this leaves you holding

So the next time something feels like just the way things are — the free tier, the open API, the platform that costs nothing, the data that flows without asking — it’s worth a second look. Is this a fact, or a default? Is it a law of the world, or a knob, and if it’s a knob, whose hand is on it?

You won’t always be able to see the hand. That’s the humbling part. Most of the arrangements you depend on were set by people you’ll never meet, for reasons that had nothing to do with you, and they hold only as long as they suit the setter. Seeing that doesn’t give you control. It gives you something smaller and more useful: the memory that the ground you’re standing on was poured by someone, and can be poured again.

03 · Lab · your turn

Whose Switch Are You On

Build a product on free defaults, then watch each one flip to feel how much of your ground belongs to someone else.

04 · Hope · carry this

When the free deal that held the web together finally got questioned, it turned out publishers could be paid and readers still served — proof that an arrangement can be reopened and rebuilt fairer, not just broken. The defaults we inherit were set by people, which means people can set better ones.