There is no way you will ever run the 405bn parameter model on consumer hardware.
For all the smaller models, a M3 MacBook pro with maxed out RAM works incredibly well. The M3 chip architecture makes it so that all the RAM is available to the GPUs as well, so you can get up to 96GB of RAM.
I can run all the models with ollama with crazy speed.
mbrochh / Martin
npub1c8…5h5s7
2024-07-25 21:09:59
in reply to nevent1q…4s88
Author Public Key
npub1c80wmfpzc7dkghh93kxtrwpe5gdztynvnk2vy93ge68zmzwrm0yqq5h5s7Published at
2024-07-25 21:09:59Event JSON
{
"id": "a21a25e322fff081a99017b938d38fe77773471dd61ca3dd5256b623068eb99f",
"pubkey": "c1deeda422c79b645ee58d8cb1b839a21a25926c9d94c21628ce8e2d89c3dbc8",
"created_at": 1721941799,
"kind": 1,
"tags": [
[
"e",
"5d54a060fd0b988e59c401a9bfa190659a54c4be9714de8bdb2d92a4ca3b4cd2",
"wss://nostr.mom/",
"root"
],
[
"e",
"dc326302c8bf0688debea7f434edabd7ab472e89510e522e2e98629225780c5e",
"wss://nostr-pub.wellorder.net",
"reply"
],
[
"p",
"46fcbe3065eaf1ae7811465924e48923363ff3f526bd6f73d7c184b16bd8ce4d"
]
],
"content": "There is no way you will ever run the 405bn parameter model on consumer hardware.\n\nFor all the smaller models, a M3 MacBook pro with maxed out RAM works incredibly well. The M3 chip architecture makes it so that all the RAM is available to the GPUs as well, so you can get up to 96GB of RAM.\n\nI can run all the models with ollama with crazy speed.",
"sig": "55a3937f81e45b559836e2d090a1d8646eaa135a2dc832020732e1295252851eb8a1ace704dadbc4ee6e635e35e3a4789fd530ede762829a2810662d989ff235"
}