The dispute between Cloudflare and AI search company Perplexity has intensified, with Cloudflare accusing Perplexity of employing tactics to bypass website restrictions. Cloudflare claims Perplexity is engaging in ‘stealth crawling’ by masking its web crawler’s identity when blocked from accessing websites.
This accusation builds on earlier concerns regarding Perplexity’s data scraping practices, including alleged circumvention of paywalls and robots.txt directives. Perplexity CEO Aravind Srinivas previously attributed such activities to third-party crawlers.
Cloudflare asserts that its customers have reported ongoing access by Perplexity bots despite implementing blocking measures. Their investigation revealed that Perplexity initially identifies its crawlers as “PerplexityBot” or “Perplexity-User.” However, when blocked, it allegedly changes its user agent to mimic Google Chrome on macOS. Further, Cloudflare alleges that Perplexity rotates IP addresses and switches autonomous system networks (ASN) to further evade detection. Cloudflare says that they observed this activity across a huge number of domains and millions of requests per day. In response, Perplexity has dismissed Cloudflare’s report as a “publicity stunt,” stating that “there are a lot of misunderstandings in the blog post.” Cloudflare has since removed Perplexity from its list of verified bots and is actively blocking its alleged evasion attempts.