Raison d'État on Nostr: I use llama.cpp, its the fastest llm server I've found for my hardware. Ollama is the ...

I use llama.cpp, its the fastest llm server I've found for my hardware.

Ollama is the easiest, but you need a recent GPU.

Transformers works well if you're writing your own harness.

And I hear good things about vLLM, especially at scale.

For getting the LLMs themselves, Github and Hugginface have more available for free download than you could ever test...