How to Block AI Crawlers and Protect Your Website

AI-powered tools like ChatGPT, Perplexity, and Claude are changing how users consume information. But there’s a hidden cost for publishers: many of these AI platforms rely on crawlers that scrape websites without sending traffic back. For site owners, this means wasted server resources, lost ad impressions, and content being reused in AI answers without credit.

To address this, Cloudflare introduced AI Audit (Beta)— a new feature that lets you analyze and block known AI crawlers. In this guide, we’ll explain what AI Audit is, why it matters, and how to safely block AI bots without affecting your SEO.

What is Cloudflare AI Audit?

AI Audit is a Cloudflare security feature that automatically detects traffic from AI crawlers. These include:

• GPTBot / ChatGPT-User (OpenAI)
• Bytespider (ByteDance / TikTok)
• ClaudeBot / Claude-User (Anthropic)
• PerplexityBot / Perplexity-User (Perplexity AI)
• Meta-ExternalAgent (Meta AI crawler)
• CCBot (Common Crawl dataset)
• DuckAssistBot (DuckDuckGo AI)
• Others like MistralAI-User, ProRataInc, Timpibot

AI Audit shows you:

• Total requests from each crawler
• Allowed vs Blocked requests
• Top paths crawled (like /wp-login.php, /wp-json, or even your articles)

Why You Should Block AI Crawlers

Blocking AI scrapers protects your site in several ways:

• Preserves hosting resources (CPU, RAM, PHP workers).
• Prevents AdSense revenue loss from wasted bot impressions.
• Keeps your content from being used in AI training without credit.
• Reduces security risks by limiting automated probes on login and API paths.

Importantly, blocking AI crawlers does not hurt SEO — as long as you continue allowing Googlebot, BingBot, and other legitimate search engines.

How to Block AI Crawlers in Cloudflare

1. Use AI Audit Toggle (Easy Method)

• Log in to Cloudflare dashboard.
• Go to Security → AI Audit.
• You’ll see a list of crawlers (GPTBot, PerplexityBot, etc.) with Allow/Block buttons.
• Toggle unwanted AI crawlers to Block.

This instantly creates a managed firewall rule in the background.

2. Use a Custom Firewall Rule (Advanced Method)

If you prefer manual control, you can create a custom rule under Security → WAF → Custom Rules.

Example rule:

(http.request.uri.path ne "/robots.txt") and 
((http.user_agent contains "Bytespider") or 
 (http.user_agent contains "CCBot") or 
 (http.user_agent contains "ChatGPT-User") or 
 (http.user_agent contains "Claude-SearchBot") or 
 (http.user_agent contains "Claude-User") or 
 (http.user_agent contains "ClaudeBot") or 
 (http.user_agent contains "DuckAssistBot") or 
 (http.user_agent contains "GPTBot") or 
 (http.user_agent contains "meta-externalagent") or 
 (http.user_agent contains "meta-externalfetcher") or 
 (http.user_agent contains "MistralAI-User") or 
 (http.user_agent contains "OAI-SearchBot") or 
 (http.user_agent contains "Perplexity-User") or 
 (http.user_agent contains "PerplexityBot") or 
 (http.user_agent contains "ProRataInc") or 
 (http.user_agent contains "Timpibot"))

• Action: Block
• Order: Place this rule above others for best effect.

Tip: Use lower(http.user_agent) to catch case variations like GPTBot vs gptbot.

Which Crawlers to Allow vs Block

Allow (SEO-friendly search engines):

• Googlebot
• BingBot
• Applebot
• Amazonbot
• PetalBot (Huawei)

Block (AI scrapers):

• GPTBot / ChatGPT-User (OpenAI)
• ClaudeBots (Anthropic)
• Bytespider (ByteDance)
• PerplexityBots
• Meta-ExternalAgent / Meta-ExternalFetcher
• CCBot
• DuckAssistBot
• Other AI crawlers (MistralAI-User, ProRataInc, Timpibot)

This way, you stay indexed on search engines but prevent free AI scraping.

Best Practices

• Always allow Googlebot and BingBot to keep your SEO intact.
• Exclude /robots.txt and /ads.txt from firewall rules.
• Monitor Cloudflare Security → Events to confirm bots are blocked.
• Use Cloudflare Analytics to check reduction in hosting load after blocking.

Cloudflare AI Audit gives publishers a much-needed tool to take back control over their content. With one click, you can see which AI bots are crawling your site and block them instantly.

See also: Cloudflare Bot Fight Mode: How to Whitelist Good Bots and Block Bad Bots

For websites running on shared hosting or with limited resources, blocking AI crawlers not only protects your bandwidth and CPU, but also helps you focus resources on real users — the ones who engage with your content and generate revenue.

If you run a WordPress blog, tech site, or any content-driven platform, enabling AI Audit is a smart move to secure your future traffic and revenue.