RE: https://mastodon.social/@arstechnica/117429325749473121
There's a stupid easy fix for this:
➡️ RESPECT ROBOTS.TXT ⬅️
That's it. If your chatbot's user agent is in robots.txt AND robots.txt says "NO", stop immediately.
The reason that this plainly obvious mechanism isn't in place is because Big AI companies feel completely entitled to anything they can crawl.
The solution isn't to ask the model, "Pretty please?" The solution is to hardcode the prohibition about circumventing when site owners tell you "NO".
Chris J. Karr
npub1rf…5fd0m
2026-10-12 18:29:01 UTC
Author Public Key
npub1rfwngle7ycxlfsv8m8sse5r2ymsx4d6j7tppuzlyka7kmf73dyfsf5fd0mSeen on
wss://relay.momostr.pinkPublished at
2026-10-12 18:29:01 UTCEvent JSON
{
"id": "33fdb0b19cc5288cce49cf4546f722e8af5439641f571b8d279ff1f1da7b4944",
"pubkey": "1a5d347f3e260df4c187d9e10cd06a26e06ab752f2c21e0be4b77d6da7d16913",
"created_at": 1791829741,
"kind": 1,
"tags": [
[
"proxy",
"https://omgwars.com/@chris/117429353924673536",
"web"
],
[
"proxy",
"https://omgwars.com/users/chris/statuses/117429353924673536",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://omgwars.com/users/chris/statuses/117429353924673536",
"pink.momostr"
],
[
"-"
]
],
"content": "RE: https://mastodon.social/@arstechnica/117429325749473121\n\nThere's a stupid easy fix for this:\n\n➡️ RESPECT ROBOTS.TXT ⬅️ \n\nThat's it. If your chatbot's user agent is in robots.txt AND robots.txt says \"NO\", stop immediately.\n\nThe reason that this plainly obvious mechanism isn't in place is because Big AI companies feel completely entitled to anything they can crawl.\n\nThe solution isn't to ask the model, \"Pretty please?\" The solution is to hardcode the prohibition about circumventing when site owners tell you \"NO\".",
"sig": "9a0db7c1b6896782baa71e5daea56177fb46fc79470f17f8209ff2bc0f4c159ef4a9c038865cd604e9053985beee4be2d30288746392960a83997407c358b573"
}