Cloudflare is rolling out new tools to let website owners manage access for different types of AI bots. Starting September 15, 2026, Training and Agent bots will be blocked by default on pages with ads, as advertising signals a preference for human visitors. Search crawlers will still be allowed. Multi-purpose crawlers like Googlebot will be treated according to the strictest rule applied. These controls are available to users on the free plan as well. | Image: Cloudflare
The changes mark a shift from a one-size-fits-all approach to more nuanced bot management. Previously, all AI crawlers could be blocked with a single click. Now, site owners can choose between three categories: Search, Training, and Agent. Training covers data collection for AI model training, while Agent refers to bots acting on behalf of users, like ChatGPT. Search crawlers are still permitted, allowing search engines to index content. Cloudflare is pushing bot operators to separate their crawlers by purpose, as CEO Matthew Prince has previously criticized Google for bundling its search and AI crawlers.
Cloudflare is also launching BotBase, a searchable database of known bots for enterprise customers. The database shows how each bot is classified and how it uses content, whether it links to pages or reproduces them in full. Previously, all verified bots were automatically allowed, but now a bot's category determines access. The 'Verified' label alone is no longer sufficient; bot operators must prove they identify themselves honestly and don't abuse access.
Source: thedecoder
