Been happily using Ollama (https://ollama.com) for running open-source LLM models locally.
But I haven't found anything similar for voice models that runs on arm64. vLLM is what Mistral recommends to run Voxtral voice models with, but it seems to be x86+CUDA only.
Any recommendations?
Frédéric Jacobs
npub12y…0vrvk
2025-07-18 09:08:32 UTC
Author Public Key
npub12y3uyznz7wvsmyynagsnf3p3nzg8vx6g8ypqvh6sw88zxqgwhxdqu0vrvkSeen on
wss://relay.momostr.pinkPublished at
2025-07-18 09:08:32 UTCEvent JSON
{
"id": "3a7eb38f19850607099645f657943f9479042acd27b9fdf6c90122afd1942bbc",
"pubkey": "5123c20a62f3990d9093ea2134c4319890761b483902065f5071ce23010eb99a",
"created_at": 1752829712,
"kind": 1,
"tags": [
[
"proxy",
"https://mastodon.social/@fj/114873448056411096",
"web"
],
[
"proxy",
"https://mastodon.social/users/fj/statuses/114873448056411096",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mastodon.social/users/fj/statuses/114873448056411096",
"pink.momostr"
],
[
"-"
]
],
"content": "Been happily using Ollama (https://ollama.com) for running open-source LLM models locally. \n\nBut I haven't found anything similar for voice models that runs on arm64. vLLM is what Mistral recommends to run Voxtral voice models with, but it seems to be x86+CUDA only. \n\nAny recommendations?",
"sig": "b976487ce10ca2462864632e924bcedfc7a79a83680f610d80b1db4654fc1a7e6973a19b5d1843f2cdfa1d084921924bb059cb3aeab4223dcedc8b3d673b9e6c"
}