Markdown edition

# Cloudflare’s AI bot defaults depend on your site’s ads and existing settings

> Cloudflare offers new ad-supported domains a no-training setting and an agent block on ad pages. Existing domains follow their migrated choices; owners can check all three controls.

By BIG CHANGE Editorial

Published: 2026-10-18T21:03:34.849Z
Updated: 2026-10-18T21:03:34.849Z
Canonical: https://bigchange.ai/blog/cloudflare-ai-search-training-agent-access-controls

![Conceptual charcoal illustration of an empty chair facing a wall-mounted display of a generic webpage with a prominent orange advertising area.](https://bigchange.ai/api/media/file/cloudflare-ad-page-concept-v2.png)
AI-generated conceptual illustration by BIG CHANGE; no real website, Cloudflare setting, crawler access or test result is depicted.

Cloudflare now gives website owners separate controls for search crawlers, model-training crawlers and AI agents. Its [September 15 rollout](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) offers new domains two different starting configurations. An ad-supported site is offered Search: Allow, Training: Disallow AI Training and Agent: Block on pages with ads. A site that does not monetize with ads is offered Allow for all three. Owners can change the recommendations during onboarding or later.

Existing Cloudflare domains follow their previous choices. A site that had left the old “Block AI Bots” switch off moves to Allow for Search, Training and Agent. One that had selected either legacy blocking option moves to Search: Allow, Training: Disallow AI Training and Agent: Block on pages with ads. Domains that had already configured the three controls retain the practical effect of those choices under Cloudflare’s [migration table](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/). The controls are available on every plan, including Free; plan eligibility does not tell an owner which policy their domain currently has.

## What the three categories cover

Cloudflare’s [July changelog](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/) defines **Search** as crawling that indexes content to answer later queries. **Agent** covers real-time activity on a person’s behalf, including chat fetch bots and browser-use agents. **Training** covers collection for model training or fine-tuning. A crawler can have more than one purpose. Cloudflare says each blocking option covers verified bots in its classification and additional unverified bots classified the same way. That is a classification-based control, not a claim that every automated browser or unidentified request will be caught.

For a domain using the ad-supported recommendation, “Block on pages with ads” applies to the Agent category on pages Cloudflare detects as serving ads. It does not describe a whole-site agent ban. Search remains allowed by that preset. The Training setting is different: September’s **Disallow AI Training** publishes a no-training preference through Bot Preference Sync and blocks training crawlers other than mixed-use crawlers Cloudflare calls Accountable. Those Accountable crawlers can continue search crawling while honoring the training preference. Cloudflare says the setting covers training-only crawlers separately, so blocking those need not shut off search.

This September arrangement refines the [July documentation](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/), which still describes a future default that blocks both Training and Agent on ad pages. The later [release announcement](https://www.cloudflare.com/press/press-releases/2026/cloudflare-helps-end-the-search-or-ai-training-tradeoff/) and Cloudflare’s detailed September migration and recommendation tables specify Disallow AI Training for an ad-supported new domain, with the ad-page block applied to Agent. The older description should not be read as a universal policy for existing customers.

“Disallow” and “Block” also have different effects on a crawler used for both search and training. Cloudflare says mixed-use crawlers account for 36.6% of verified crawler traffic on its network, though it gives no comparable figure here for individual sites. A full Training block now includes mixed-use crawlers such as Googlebot, Bingbot and Applebot, which can remove their search access too. Disallow AI Training is intended to retain search access for Accountable mixed-use crawlers while expressing the training restriction. Cloudflare notes a present limit: Microsoft has not yet shipped support for a no-training preference in `robots.txt` for Bingbot, so selecting Disallow AI Training does not automatically convey that preference to Bing today.

## What a site owner can check

In the Cloudflare dashboard, open the domain’s **Security Settings → Configure AI bot policies** and inspect the current Search, Training and Agent selections. Cloudflare’s [configuration documentation](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) describes Allow, Block and Block on pages with ads; its September account explains the additional Disallow AI Training choice and the migration of legacy settings. If the site relies on ad impressions, check whether the domain uses the ad-supported preset and whether Agent is limited to detected ad pages. If preserving conventional search access matters, check the Training choice before selecting Block for mixed-use crawlers.

These are Cloudflare’s documented controls and rollout rules. BIG CHANGE did not change a Cloudflare account or measure bot traffic on a live site.

## The big change

Cloudflare has separated three uses of automated access into domain-level policies. For new ad-supported domains, its September recommendation keeps search open, expresses a no-training preference and blocks classified agents on detected ad pages. Existing owners need to read their own migrated settings before drawing conclusions about access.

## Sources & further reading

- [Cloudflare, “Have it both ways: stay discoverable in search while disallowing AI training”](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) (September 15, 2026). The detailed rollout source: recommendation and migration tables, mixed-use crawler behavior, and the Bing limitation.
- [Cloudflare, “Cloudflare Helps End the Search-or-AI-Training Tradeoff”](https://www.cloudflare.com/press/press-releases/2026/cloudflare-helps-end-the-search-or-ai-training-tradeoff/) (September 15, 2026). The launch announcement distinguishes ad-supported and other new sites.
- [Cloudflare Developers, “New options to manage AI traffic”](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/) (July 1, 2026). The original three-category launch and earlier default description.
- [Cloudflare Docs, “Block AI Bots”](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) (page marked last updated July 1, 2026). Definitions, classified verified and unverified bots, and dashboard location; its pre-rollout default wording differs from September’s detailed policy.

## Sources

- [Cloudflare, Block AI Bots (last updated 2026-07-01)](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/) — Primary configuration documentation, stamped July 1: category definitions, verified and unverified classifications, ad-page detection and dashboard path. Its prospective default wording predates the September rollout and is superseded by the later detailed account.
- [Cloudflare, AI traffic controls announcement (2026-09-15)](https://www.cloudflare.com/press/press-releases/2026/cloudflare-helps-end-the-search-or-ai-training-tradeoff/) — Primary launch announcement: new ad-supported sites get Search allowed, Training disallowed and Agent blocked on ad pages; other new sites allow all three.
- [Cloudflare Developers, New options to manage AI traffic (2026-07-01)](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/) — Original three-category launch and prospective September defaults, retained for chronology rather than treated as the final rollout rule.
- [Cloudflare, Have it both ways: stay discoverable in search while disallowing AI training (2026-09-15)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) — Controlling detailed rollout record: new-domain recommendations, existing-domain migration, Disallow versus Block for mixed-use crawlers and Bingbot limitation.