Cloudflare will block mixed-use AI crawlers by default on ad-supported pages from Sept. 15
Crawlers that blend search, agent use, and training get blocked unless the site owner changes the setting, TechCrunch reported.
Starting Sept. 15, 2026, Cloudflare's default settings will block "mixed-use" crawlers from pages that host ads, TechCrunch reported on July 1. Crawlers that combine search, agents, and training are blocked unless site owners opt out of the default.
“We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training”
Matthew Prince, via TechCrunch
Sources
Spotted an error? See our corrections policy.
Related
Expert network Guidepoint launches Guidepoint Labs to sell expert data to AI labs
The unit ran in stealth for more than six months and says it already works with frontier labs. It draws on Guidepoint's network of more than 2 million advisors.
Humyn Labs raises $47 million to record real-world human data for robotics AI
Accel led $26 million in equity and Lighthouse Canton led $21 million in venture debt. The Bengaluru startup films people doing everyday tasks in more than 20 countries and sells the labeled footage to AI and robotics developers.
Mercor hires Jason Wei, formerly of Meta's TBD lab, as chief scientist
The expert-data company announced the hire a week after its $750 million Series D. Wei created benchmarks including SimpleQA.
Secludy's Who Sells Data register lists 208 AI training data and RL environment vendors
The register, last updated Nov. 11, is run by Secludy AI, which invites readers to book time with its CEO, Ben Cerchio.