captjack on Nostr: GPU ACCELERATION ✓ You have an NVIDIA or AMD GPU ✓ You want 3-10x faster ...

GPU ACCELERATION

✓ You have an NVIDIA or AMD GPU
✓ You want 3-10x faster inference for local LLMs

NVIDIA GPU

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit nvidia-cuda-toolkit nvidia-docker2
sudo systemctl restart docker


AMD GPU

# 1. Add ROCm GPG key
wget https ://repo. radeon.com/rocm/rocm.gpg.key -O - | \
gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null

# 2. Add repository
source /etc/os-release
echo "deb [signed-by=/etc/apt/keyrings/rocm.gpg] https ://repo. radeon.com/amd-container-toolkit/apt/ $VERSION_CODENAME main" \
| sudo tee /etc/apt/sources.list.d/amd-container-toolkit.list

# 3. Install
sudo apt update && sudo apt install -y amd-container-toolkit

# 4. Configure Docker
sudo amd-ctk runtime configure
sudo systemctl restart docker