AI search fight: Cloudflare and Perplexity clash over crawling

Cloudflare accused AI answer engine Perplexity of “stealth crawling,” saying it uses deceptive techniques to bypass website blocks and access content it’s been explicitly told not to touch.

• In response, Perplexity said Cloudflare has a fundamental misunderstanding of how AI assistants work and accused the company of either publicity-seeking or technical incompetence.

The big picture. Cloudflare said Perplexity uses declared bots when it can, but switches to “stealth crawling” when blocked. That includes mimicking normal browser behavior, rotating IPs, and ignoring robots.txt rules (tactics that can be associated with scrapers and bad actors).

• Cloudflare tested this by setting up honeytrap sites and found Perplexity answering questions using content it shouldn’t have been able to access.
• Perplexity insisted its requests are made on behalf of users, not as preemptive crawling. The company says these are real-time fetches, akin to what a browser or email client does, and claims Cloudflare mistook its behavior for something it wasn’t.

Why we care. If AI assistants can sidestep robots.txt by posing as browsers, brands, creators, and publishers lose control over how and when their content is used. That breaks the old deal between search engines and websites.

What’s next. Cloudflare said it’s already blocking the behavior in question and expects Perplexity’s tactics to change in response. It’s calling for standardization of bot behavior through IETF (the Internet Engineering Task Force) and other policy efforts.

• Perplexity, meanwhile, is doubling down on its identity as an agentic AI platform and says it shouldn’t be governed by rules designed for traditional web crawlers.

The blog posts. You can view the full back and forth here:

• Cloudflare: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
• Perplexity: Agents or Bots? Making Sense of AI on the Open Web

Read more at Read More