Home RAG assistant for embedded-systems docs on Proxmox

What they built

The author wanted a local assistant that answers embedded-work questions from their own reference material, without any hosted service.

It started on the GPUs they already had (6 GB cards) to prove the stack, then moved to an RTX 5060 Ti 16 GB, which made bigger and MoE models practical.

Setup

• Debian VM on Proxmox, GPU via VFIO passthrough, services in Docker
• Ollama plus llama.cpp as a second OpenAI-compatible backend
• Open WebUI front end
• BGE-M3 embeddings + bge-reranker-v2-m3; SearXNG for web search
• GPUs: GTX 1660 Ti 6 GB → GTX 1060 6 GB → RTX 5060 Ti 16 GB

Results & numbers

• 14B coding model with 32K context in ~15 GB VRAM (spills out around 43K)
• Gemma 4 26B-A4B at ~35 tokens/s at 16K context
• 35B-A3B Qwen with CPU-offloaded experts: 16 → 28.5 tokens/s after setting no-mmap

Gotchas

• Chipset-fed slot plus Resizable BAR broke host boot; fixed by moving the GPU to the CPU slot and updating the BIOS
• Regenerate the NVIDIA CDI spec after any GPU or driver change, or containers silently lose the GPU
• For RAG, qwen2.5-coder:7b beat qwen3:4b, whose thinking mode wasted tokens
• An apparent vision hallucination was really an Ollama image ingestion bug

“With mmap on, the expert weights left in system RAM were being faulted in from disk”— Ground Bounce

Do it yourself

1. Create a Debian VM with GPU passthrough (q35, cpu: host, ballooning off, hugepages)
2. Run Ollama, Open WebUI and SearXNG in Docker
3. Use BGE-M3 + bge-reranker-v2-m3 for hybrid retrieval
4. Chunk technical docs at ~2,000-2,500 chars with 300 overlap
5. Find your real context limit empirically; set no-mmap for MoE CPU offload

nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml