If you are running a Mac this is a little easier, btw because of the unified bus. A Mac shares the bus and RAM across the GPU and CPU, which makes running small and medium-sized models both performant and possible with Ollama.
These models are GPU and RAM hungry between the model size and the context size.
The best video I have seen to somewhat fit it in my head is: https://www.youtube.com/watch?v=QfFRNF5AhME
Jeff Triplett
npub1ns…8ahke
2025-01-27 14:13:08 UTC
in reply to nevent1q…jckv
Author Public Key
npub1nsp3hg75ge84dn5fttlkpxpxnj0dfl697239dtyz4js3mdw2jllqf8ahkePublished at
2025-01-27 14:13:08 UTCEvent JSON
{
"id": "a487adfd1c546948331beea316da96071f5a79a3023e857e63c5dabaed81daa5",
"pubkey": "9c031ba3d4464f56ce895aff6098269c9ed4ff45f2a256ac82aca11db5ca97fe",
"created_at": 1737987188,
"kind": 1,
"tags": [
[
"proxy",
"https://mastodon.social/@webology/113900728387583057",
"web"
],
[
"e",
"7ef35cfb8ecfa7a35718907097b76098ce2e813b82a77d2c6d2ea26a02be57b1",
"",
"reply",
"bca276c2efbc77f202037845428983726efbee92fd4d8ab58d6c07cf9288290d"
],
[
"p",
"bca276c2efbc77f202037845428983726efbee92fd4d8ab58d6c07cf9288290d"
],
[
"p",
"9c031ba3d4464f56ce895aff6098269c9ed4ff45f2a256ac82aca11db5ca97fe"
],
[
"e",
"7347e64e2e739da6fe34c2f85d5f05fca94972236cb66429dd322002d969d23f",
"",
"root",
"0cf00a06e28525f57ff967649bfb9b28aa75a4909d7c1f7d97fb94ceb1c86c11"
],
[
"p",
"0cf00a06e28525f57ff967649bfb9b28aa75a4909d7c1f7d97fb94ceb1c86c11"
],
[
"proxy",
"https://mastodon.social/users/webology/statuses/113900728387583057",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mastodon.social/users/webology/statuses/113900728387583057",
"pink.momostr"
],
[
"-"
]
],
"content": "If you are running a Mac this is a little easier, btw because of the unified bus. A Mac shares the bus and RAM across the GPU and CPU, which makes running small and medium-sized models both performant and possible with Ollama. \n\nThese models are GPU and RAM hungry between the model size and the context size. \n\nThe best video I have seen to somewhat fit it in my head is: https://www.youtube.com/watch?v=QfFRNF5AhME",
"sig": "ce99085abbad40efe6e5c71cbf27cbec269da1a9f38adea93228f5fa7759e9172ffef0a4173dc9962241e66c55aa03c8fbb0789c37e6ddb36e71f83eb5622daf"
}