nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqkr8yrx4h6t40jvsx6lll4wxkelf582t7jxqk3x98rcr23mufyqhqawhaal (nprofile…haal)
They speak about Llama 3. To fit 70b into 24 GB of VRAM, you have to quantizize it down to IQ2_S, which means 2 bit. You can achieve about 10 tokens per second with that kind of quantization.
TUXEDO
npub14z…s9lef
2025-03-20 03:53:01 UTC
in reply to nevent1q…6098
Author Public Key
npub14zhkws7j25rpy2pktdrawq97660dynvzrxya9588ttfrptplkplq6s9lefSeen on
wss://relay.ditto.pubPublished at
2025-03-20 03:53:01 UTCEvent JSON
{
"id": "1081067412e569b881d8570a95d9b0bc4e9949f8aaa1d83a4957816a6a3bc7b0",
"pubkey": "a8af6743d255061228365b47d700bed69ed24d821989d2d0e75ad230ac3fb07e",
"created_at": 1742442781,
"kind": 1,
"tags": [
[
"p",
"b0ce419ab7d2eaf93206d7fffab8d6cfd343a97e91816898a71e06a8ef89202e",
"wss://relay.mostr.pub"
],
[
"p",
"6c446370a6b3fde53e3a0f49e7826e53755fb033a7c0f9aed1a21ddf17d7913f",
"wss://relay.mostr.pub"
],
[
"e",
"040241f4c8ac49cd52c9eb4d8edad6d1963c2857a87cb93e5c406dbafe0fe5ec",
"wss://relay.mostr.pub",
"reply"
],
[
"proxy",
"https://linuxrocks.online/users/tuxedocomputers/statuses/114192730152053333",
"activitypub"
]
],
"content": "nostr:nprofile1qy2hwumn8ghj7un9d3shjtnddaehgu3wwp6kyqpqkr8yrx4h6t40jvsx6lll4wxkelf582t7jxqk3x98rcr23mufyqhqawhaal \nThey speak about Llama 3. To fit 70b into 24 GB of VRAM, you have to quantizize it down to IQ2_S, which means 2 bit. You can achieve about 10 tokens per second with that kind of quantization.",
"sig": "1beaa5be4ea7ab541271ec86a4b5b1ff8b2ccaacb150bd155769eda07553c6b03aea4c5d974c4d713b6096457e8ca64cf5462c0bad26e80a122968b0fafa5ce5"
}