HelpfulGiraffe on Nostr: Web scrapers have been ignoring the robots.txt file and using a rotating proxy since ...

Web scrapers have been ignoring the robots.txt file and using a rotating proxy since they came out. I would have been surprised if perplexity wasn’t doing that

quoting
nevent1q…qnph

AI site Perplexity uses “stealth tactics” to flout no-crawl edicts, Cloudflare says - AI search engine Perplexity is using stealth bots and other ... - https://arstechnica.com/information-technology/2025/08/ai-site-perplexity-uses-stealth-tactics-to-flout-no-crawl-edicts-cloudflare-says/ #robots.txt #security #scraping #biz⁢ #ai