And then there's evaluation, or lack thereof:
Google is advertizing Gemini as an everything machine---a general purpose model that can be used in many different ways. In other words: sthg that cannot be evaluated, since it doesn't have a specific purpose.
What stands in for evaluation are "benchmarks", but these benchmarks lack construct validity. What are they supposed to be measuring? What shows that they do measure that? How does that relate to the intended use case of the technology?
/5
Prof. Emily M. Bender(she/her) /
npub1z0…0pp3u
2023-12-08 05:09:00
in reply to nevent1q…pwwj
Author Public Key
npub1z0kfl4g93gvv6ztazp0adm6rwk0r04v3tvwqrmfk4ncw7k37du4qk0pp3uPublished at
2023-12-08 05:09:00Event JSON
{
"id": "7d8d187ecb6db456cd1fa1c2c40085e948d3647a43b03d0bdc86149b77020bf7",
"pubkey": "13ec9fd5058a18cd097d105fd6ef43759e37d5915b1c01ed36acf0ef5a3e6f2a",
"created_at": 1702012140,
"kind": 1,
"tags": [
[
"e",
"b5531462657edb599d65941acf1c9f62d426e372a47459f3572b076c35814155",
"wss://relay.mostr.pub",
"reply"
],
[
"proxy",
"https://dair-community.social/users/emilymbender/statuses/111543067628522918",
"activitypub"
]
],
"content": "And then there's evaluation, or lack thereof: \n\nGoogle is advertizing Gemini as an everything machine---a general purpose model that can be used in many different ways. In other words: sthg that cannot be evaluated, since it doesn't have a specific purpose. \n\nWhat stands in for evaluation are \"benchmarks\", but these benchmarks lack construct validity. What are they supposed to be measuring? What shows that they do measure that? How does that relate to the intended use case of the technology?\n/5",
"sig": "34cf13d4f6796fcfa604dd70a27f26603d9143197e3e61cb147526ed76eb427f7f7ecb4926ac0a8a3303f00e1796d6620b9e58cf8b57ce7155610db47173cd10"
}