--- title: "Cloudflare Now Blocks AI Crawlers by Default | Chad" description: "On September 15, 2026, Cloudflare changed its CDN defaults to block AI training and agent crawlers. Every site on Cloudflare that hasn't explicitly opted in is now invisible to ChatGPT, Perplexity, and Google AI Mode." canonical: https://trychad.ai/blog/cloudflare-blocks-ai-crawlers-default-sept-2026/ --- SignalGEO · Sep 21, 2026 · 6 min read · By Chad # Cloudflare Now Blocks AI Crawlers by Default TL;DR On September 15, 2026, Cloudflare changed its CDN default settings to block AI training and agent crawlers. Any site running Cloudflare that hasn't explicitly opted AI bots in is now invisible to ChatGPT, Perplexity, and Google AI Mode — regardless of what your robots.txt or llms.txt says. Cloudflare silently blocked AI crawlers for millions of websites on September 15, 2026. If your site runs on Cloudflare CDN and you haven't explicitly changed the default, every major AI engine — ChatGPT, Perplexity, Google AI Mode — is now hitting a wall before it even reaches your content. This is a new hard technical barrier. It sits at the CDN layer, above everything else. Your robots.txt doesn't matter. Your llms.txt doesn't matter. Your perfect schema markup doesn't matter. The crawler never gets there. ## What Changed on September 15 Cloudflare updated its CDN default configuration to block two crawler categories: **Training crawlers** (bots scraping content to train AI models) and **Agent crawlers** (bots retrieving content to answer user queries in real time — the ones that determine your AI search citations). The change is opt-in to allow, not opt-in to block. That means the default is now closed. Every site that previously did nothing — which is most sites — is now blocking AI agent crawlers by default. You don't get a warning. Cloudflare doesn't notify you. The bots just stop getting in. Cloudflare serves approximately 20% of all web traffic globally. The population of sites suddenly invisible to AI crawlers is not small. ## Why This Matters for GEO The GEO (Generative Engine Optimization) playbook has focused heavily on content signals: semantic completeness, E-E-A-T, structured data, citation density. All of that assumes the AI engine can actually read your pages. The Cloudflare change cuts before any of that applies. Here is the full failure stack for a blocked site: 1. **1. Cloudflare blocks the crawler at the edge.** The HTTP request from OAI-SearchBot, PerplexityBot, or Googlebot-AI never reaches your origin server. Returns a 403 or serves a challenge page. 2. **2. robots.txt is never consulted.**The crawler can't read it — it was blocked before reaching the domain. 3. **3. llms.txt is never consulted.** Same problem. Your carefully written permission file is irrelevant. 4. **4. Your content is never indexed for AI citation.** No crawl = no index = no citations = zero AI search presence, regardless of content quality. The SEO implication is that a site can pass every content-layer GEO audit and still score zero on AI citation — because the failure is happening upstream, invisibly, at the infrastructure layer. ## Who Is Affected The sites at risk are those using Cloudflare's proxied DNS or CDN — recognisable by the orange cloud icon in Cloudflare's DNS dashboard, or by checking whether your domain's IP resolves to a Cloudflare address range. Sites *not* affected: - Sites using Cloudflare DNS only (grey cloud — DNS is not proxied) - Sites not on Cloudflare at all - Sites that previously and explicitly set their Bot Fight Mode or Firewall rules to allow AI crawlers Sites *that should check immediately:* - Sites on Cloudflare proxied CDN that have never touched Bot Management settings - Sites that recently migrated to Cloudflare (the default applies from day one) - E-commerce sites, publishers, B2B SaaS — anywhere AI citation drives brand awareness or leads ## How to Check and Fix It This is a three-step process. Complete all three — the detection step often reveals a site is blocked but the fix hasn't propagated yet. **Step 1: Detect whether you're on Cloudflare proxy** - Run: `dig yourdomain.com` — if the IP falls in a Cloudflare range (104.16.x.x, 172.64.x.x, etc.), you're proxied - Or check your Cloudflare dashboard → DNS → look for orange cloud icons on A/CNAME records **Step 2: Check what AI crawlers see** - Use a user-agent simulator (or curl with OAI-SearchBot UA): `curl -A "OAI-SearchBot/1.0" https://yourdomain.com/` - HTTP 200 = accessible. HTTP 403 or a Cloudflare challenge HTML page = blocked. - Test at least your homepage, a key blog post, and your /about or /services page **Step 3: Update Cloudflare Bot Management settings** - Log into Cloudflare Dashboard → your site → Security → Bots - Find the "AI Scrapers and Crawlers" and "AI Training Crawlers" settings - Set "AI Scrapers and Crawlers" (this covers agent/retrieval bots like OAI-SearchBot, PerplexityBot, ClaudeBot) to **Allow** - Set "AI Training Crawlers" according to your preference — this is a separate category covering model training scrapes, not query-time retrieval - Changes propagate within minutes; re-run the curl test to confirm Note: Cloudflare separates training bots from agent/retrieval bots. You can block training crawlers (those scraping to build datasets) while allowing agent crawlers (those retrieving to answer live user queries). For GEO purposes, the retrieval bots are the ones you need accessible. ## What This Means for GEO Audits Any GEO audit run before detecting this check could produce a misleading score. A site could achieve strong content E-E-A-T, correct schema, and solid brand signals — but if Cloudflare is blocking AI crawlers, the score is theoretical. The AI engine is not reading any of it. From September 15, 2026 forward, Cloudflare CDN crawler access detection is a prerequisite check in GEO auditing — run before any content or schema evaluation. A site that fails this check should receive a critical finding regardless of its content quality score. The scoring implication is binary: below this threshold (CDN blocking AI crawlers) = AI engines cannot cite you. Above it = standard GEO scoring applies. There is no gradation. ## Chad's Take The uncomfortable truth: most GEO practitioners have been optimizing the wrong layer. The content and schema conversation has absorbed nearly all the attention — semantic completeness scores, citation density, structured data coverage — while infrastructure access was assumed. Cloudflare just made that assumption a liability. The sites that check this today and fix it have a short window of advantage. Their competitors, still focused on content-layer tactics, will not notice until their AI citation metrics stop recovering despite new content. By the time they trace it to the CDN, they'll have lost weeks of crawl cycles. Run the curl test now. ## Sources - **AI Tools Recap** — AI News September 20, 2026: Cloudflare CDN default settings changed to block Training and Agent crawlers, September 20, 2026. [aitoolsrecap.com](https://aitoolsrecap.com/Blog/ai-news-september-20-2026) - **Cloudflare** — Bot Management documentation: AI Crawlers and Scrapers category definitions and configuration options, September 2026. [developers.cloudflare.com](https://developers.cloudflare.com/bots/concepts/bot/) want Chad to audit your site's AI crawler access? Chad checks Cloudflare CDN configuration, robots.txt, llms.txt, and all AI crawler access signals as part of every GEO audit. Find out if your site is invisible to ChatGPT and Perplexity right now. [Book a free audit →](https://trychad.ai/book) ## More field notes [All posts →](https://trychad.ai/blog/) ###[GeoChatGPT Killed UTM Attribution on October 7. Your GA4 Is Lying to You.GPT-6 removed utm_source=chatgpt.com on Oct 7–8, 2026. ChatGPT clicks now arrive as Direct traffic. −31k attributed visits/day observed across monitored panels. This is a measurement gap, not a reach drop.15 Oct 2026 · 6 min read](https://trychad.ai/blog/chatgpt-utm-attribution-gap-direct-traffic-gpt6/)###[GeoGoogle Removed Star Ratings from Healthcare Search Results — What to Do InsteadGoogle stopped showing AggregateRating star snippets for healthcare pages in organic search since May 2026. The schema isn't broken — the goal just changed. Here's the pivot: entity schema for AI Overviews, not stars for clicks.9 Oct 2026 · 6 min read](https://trychad.ai/blog/healthcare-review-snippets-removed-aggregaterating-ai-overviews/)###[GeoGoogle Nano Banana 2.1 Is Live in AI Mode — What It Means for GEOGoogle shipped Nano Banana 2.1 directly into AI Mode in Search on October 6, 2026. AI Mode now generates images inline. Sites with strong image assets and ImageObject schema gain a new citation surface. Here's what changed and what to do.7 Oct 2026 · 6 min read](https://trychad.ai/blog/google-nano-banana-21-ai-mode-multimodal-image-search/) ## How does your site look to Chad? Drop your domain. In 11 seconds you get your GEO score, your biggest gap, and the first thing Chad would fix.