Kevin Boyd (he/him) 🇨🇦 on Nostr: AI crawlers finally started hitting my site hard enough for me to notice (despite a ...
AI crawlers finally started hitting my site hard enough for me to notice (despite a robots file disallowing ALL indexing from ANYONE). I had to block 8.210.0.0/16 and 47.(239|242|243).0.0/16 ... the ua did not specify who was doing the crawling, it mimicked standard browser UAs.
Published at
2025-08-23 22:04:33 UTCEvent JSON
{
"id": "2fd9312ba6f28c15bef811520e3090a41e456fbc9908963ad4d6b8a5cb427ab5",
"pubkey": "76bf383d82f5c3f0c0dd6cca201ac0e991cc80c9abace93e7f781c4dca487920",
"created_at": 1755986673,
"kind": 1,
"tags": [
[
"proxy",
"https://phpc.social/users/kboyd/statuses/115080342657553034",
"activitypub"
],
[
"client",
"Mostr",
"31990:6be38f8c63df7dbf84db7ec4a6e6fbbd8d19dca3b980efad18585c46f04b26f9:mostr",
"wss://relay.mostr.pub"
]
],
"content": "AI crawlers finally started hitting my site hard enough for me to notice (despite a robots file disallowing ALL indexing from ANYONE). I had to block 8.210.0.0/16 and 47.(239|242|243).0.0/16 ... the ua did not specify who was doing the crawling, it mimicked standard browser UAs.",
"sig": "cfd47d5b29e86b6be1acdb8cb44c4676422f26ec412194837258e0e023969fa320a50c00c78d8937a00c0482699a34e134f791714465bbac17ce3e7eea54d7c2"
}
