Tim Bray 🇨🇦 on Nostr: If you have a web site, how do you tell the AI crawlers what they can and can’t do? ...
If you have a web site, how do you tell the AI crawlers what they can and can’t do? Apparently there’s no consensus out there. IETF is trying to build something useful:
https://www.ietf.org/blog/aipref-wg/#genai #robots
Published at
2025-02-28 16:55:35 UTCEvent JSON
{
"id": "f961a5f7b996d1af25598fb1ef5ee7f111b1cf59e8005ccf9e147dfe49a6bbc3",
"pubkey": "973c779cb5e72ff0b6f0f47f8e86aee1face1c21b80ae0d9bc50968aa717e4b8",
"created_at": 1740761735,
"kind": 1,
"tags": [
[
"t",
"genai"
],
[
"t",
"robots"
],
[
"proxy",
"https://cosocial.ca/users/timbray/statuses/114082561080569089",
"activitypub"
]
],
"content": "If you have a web site, how do you tell the AI crawlers what they can and can’t do? Apparently there’s no consensus out there. IETF is trying to build something useful: https://www.ietf.org/blog/aipref-wg/\n\n#genai #robots",
"sig": "78bbf83a676cd99d508174faece4cc89a682c0bf4c34d720bf46f7597cb179a40f529d9422818bd6ee2de215103084c9975ef84e51d5484797e7b60bfc69e45b"
}
