AI scraping volume is centralizing the web by making "hosting a website" as difficult as "hosting your own email".
even if you are fine having your site crawled, the scrapers are so aggressive that you basically need to use a CDN or tool like Anubis or be able to serve millions of req/day
bryan newbold
npub1tu…tf77h
2025-08-14 16:52:22 UTC
Author Public Key
npub1tull74k2f75pwlc39yg2xn6tve9k6g854yjxafuu4x9rqggcnwas6tf77hPublished at
2025-08-14 16:52:22 UTCEvent JSON
{
"id": "1686db4648c564559af609e62ec11d7fe0619ed522949b570eef9f098c1f64a9",
"pubkey": "5f3fff56ca4fa8177f112910a34f4b664b6d20f4a9246ea79ca98a3021189bbb",
"created_at": 1755190342,
"kind": 1,
"tags": [
[
"proxy",
"https://bsky.brid.gy/convert/ap/at://did:plc:44ybard66vv44zksje25o7dz/app.bsky.feed.post/3lwesgi5h7s2x",
"activitypub"
],
[
"client",
"Mostr",
"31990:6be38f8c63df7dbf84db7ec4a6e6fbbd8d19dca3b980efad18585c46f04b26f9:mostr",
"wss://relay.mostr.pub"
]
],
"content": "AI scraping volume is centralizing the web by making \"hosting a website\" as difficult as \"hosting your own email\".\n\neven if you are fine having your site crawled, the scrapers are so aggressive that you basically need to use a CDN or tool like Anubis or be able to serve millions of req/day",
"sig": "075370933421fe90d546154ba432338e43c94519cd7bdb1cde6f3bdd83d129d6c5a9263f49f4c1431dd2ad4a0c9bc0c65531f0473adbd0f0e7b5eb79ea1e22c4"
}