Jamesob's GitHub repository outlines a step‑by‑step guide for deploying state‑of‑the‑art large language models on local hardware, including recommended system components and Docker‑based deployment configurations.

The article discusses cost thresholds, suggesting configurations around $2,000 for smaller models and $40,000 for systems capable of running models approaching the size of Claude Opus.

The guide specifies a base system built around a last‑generation EPYC processor and DDR4 memory sourced from second‑hand markets, paired with four NVIDIA RTX PRO 6000 workstation GPUs providing 384 GB of VRAM. An independent PCIe Gen4 switch enables peer‑to‑peer communication between GPUs, reducing latency compared with routing traffic through the motherboard’s PCIe root complex. Custom BIOS settings—including PCIe link width, forced Gen4 operation, disabling ASPM and SR‑IOV, and enabling Resizable BAR—are described as necessary for stable operation. Kernel parameters such as iommu=off and NCCL configuration prevent NCCL hangs in multi‑GPU environments, while ACS is disabled via a runtime script to keep peer‑to‑peer traffic within the switch fabric. Power management limits each GPU to 350 W, allowing the system to operate on a standard 110 V circuit. Performance metrics report line rates of 27.5 GB/s upstream and 50.4 GB/s bidirectional with sub‑microsecond latency. Docker‑compose files in the runners directory provide ready‑to‑run serving configurations for models such as GLM‑5.2‑594B and Whisper‑large‑v3 for speech‑to‑text, with reported throughput of approximately 80 tokens per second for the 594 B model. Additional tools include a GPU speed and latency benchmark script. The author notes that model weights are stored locally on a ZFS‑replicated filesystem and accessed via read‑only mounts in individual Docker containers.

The guide acknowledges alternative approaches, such as using a DGX Spark cluster with greater VRAM, and states that the presented configuration reflects personal experience rather than a definitive recommendation.