Cloudflare and Perplexity slug it out over alleged AI bot block evasion

Is it deceptive, standards-violating content scraping or a security and internet infrastructure company on a crusade against artificial intelligence startups? 

That question has arisen after Cloudflare published a very detailed report alleging that Perplexity AI engages in underhanded behaviour to content on sites, sparking a public row between the two companies.

Perplexity AI has struck back against the accusations, and denied it is doing anything wrong, instead questioning Cloudflare's competence when it comes to understanding what's legitimate web traffic and what isn't.

The background here is that Cloudflare has a commercial interest in bot management as a service for its customers.

It recently launched "Content Independence Day," an initiative that enables over 2.5 million websites to block AI training crawlers, a move aimed at providing content creators more control over how their material is accessed by artificial intelligence companies.

In the report, Cloudflare accused Perplexity of having obscured their bots’ crawling identities using multiple technical tricks including using different networks, faking the user-agent identifier to make it look like a web browser was accessing content, and ignoring the industry standard robots.text file to exclude automated scrapers.

Cloudflare alleged this was done so as to circumvent website administrators’ preferences to manage bot traffic, which can be substantial with tens of millions of requests a day and significantly add to data charges.

That is not the case, Perplexity insists.

Perplexity said Cloudflare had mischaracterised user-driven AI assistants as malicious bots.

"This controversy reveals that Cloudflare's systems are fundamentally inadequate for distinguishing between legitimate AI assistants and actual threats," Perplexity wrote in response.

"If you can't tell a helpful digital assistant from a malicious scraper, then you probably shouldn't be making decisions about what constitutes legitimate web traffic."

Perplexity said modern AI assistants go to relevant websites to read information needed that they don't have, to present tailored summaries to users' questions.

This is the traffic that Cloudflare said was generated by Perplexity bots ignoring webmasters' preferences to shut out AI crawlers.

Cloudflare had also confused Perplexity with up to 6 million daily requests of unrelated traffic from BrowserBase, a third party service that provides web browsing for AI agents, Perplexity alleges.

Cloudflare, meanwhile ,has de-listed Perplexity as one the vendor-declared bots for its bot management site.

It has also updated its system heuristics to ensure that blocking of the bot by customers who wish to do so is effective.

The AI company has been accused of similar behaviour in the past, with media outlets such as Forbes, Time Magazine and Wired accusing Perplexity of generating plagiarised material in its responses. Perplexity agreed last year to a revenue sharing deal with publishers last year.