RSS Feed/News Cloudflare accuses Perplexity of violating crawl rules — fair or not?

[ENXF NET]

Cloudflare claims that Perplexity has been bypassing robots.txt directives using undeclared crawlers with rotating IPs and user-agents to avoid being blocked.

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives ​

Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.

blog.cloudflare.com

Perplexity responded by saying the traffic likely came from a third-party partner (Browserbase), and emphasized that their system only accesses websites in response to direct user queries — not for autonomous...

Read more

Continue reading...