Join Nostr
2025-08-01 22:00:21 UTC
in reply to

Julian Fietkau on Nostr: Looks pretty identical to me too. It's complete guesswork, but the reason may not ...

Looks pretty identical to me too.

It's complete guesswork, but the reason may not necessarily have anything to do with the HTML code. It could be some infrastructure difference between the sites, like if one is using some proxy (Cloudflare or similar) that the other isn't, or a short-term flood of requests triggered a bot detection somewhere and blocked the crawler's IP. I've had that happen with a site that was unintentionally blocking Googlebot after it hit a firewall heuristic.