Local AI Server Build
Why
I was tired of hitting API rate limits on Claude and OpenAI during development sprints. More importantly, I wanted to experiment with open-weight models (Llama 3, Mistral) without worrying about token costs or data privacy — everything stays on my network.
Architecture / Stack
- Hardware: Used ASUS ROG Strix Z690-E motherboard, Intel i7-12700K, 128 GB DDR5 RAM
- GPU: NVIDIA RTX 3090 (24 GB VRAM) — second slot open for a companion card
- Storage: NVMe SSDs for model cache + HDD array on Unraid for checkpoint storage
- Software stack: Docker containers running Ollama, vLLM, and custom Python agent scripts
- Networking: Dedicated 10 GbE link to the main NAS for fast dataset transfers
Build Log
- Ordered parts from a mix of Newegg and used-marketplace deals (late January)
- Assembled barebones in February — first boot took three tries because I forgot to plug in the CPU cooler fan header, motherboard throttled immediately
- Installed Ubuntu Server 24.04 LTS headless with Docker + NVIDIA Container Toolkit
- Deployed Ollama as a systemd service behind a reverse proxy (Nginx on the host)
- Pulled Llama-3-8B-Instruct quantized to Q4_K_M — first inference was under 2 seconds for initial token
What Broke
The RTX 3090’s stock cooler is loud enough that my dog refused to enter the room during long training runs. Swapped to a water block mid-March, which meant disassembling everything and redoing cable management. Also learned that Docker volumes don’t survive an apt upgrade of nvidia-docker if you’re using bind mounts incorrectly — switched to named volumes across all containers.
What’s Next
- Add second GPU (waiting on a used RTX 4090 deal)
- Set up automatic model downloads when new quantized versions are released
- Build a web UI for prompt testing that sits alongside the Speaches instance
Related Projects
- Unraid Migration — stores all model checkpoints