Marketplaces

Cloudflare will block mixed-use AI crawlers by default on ad-supported pages from Sept. 15

Crawlers that blend search, agent use, and training get blocked unless the site owner changes the setting, TechCrunch reported.

Starting Sept. 15, 2026, Cloudflare's default settings will block "mixed-use" crawlers from pages that host ads, TechCrunch reported on July 1. Crawlers that combine search, agents, and training are blocked unless site owners opt out of the default.

“We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training”

Matthew Prince, via TechCrunch

Sources

  1. TechCrunch, July 1, 2026

Spotted an error? See our corrections policy.

Related