Colibrì Demonstrates Running 744B GLM‑5.2 Model on Consumer Machine

The open‑source project colibrì shows that the 744‑billion‑parameter GLM‑5.2 Mixture‑of‑Experts model can be executed on a desktop computer with 25 GB of RAM using only C code and disk‑streamed expert weights.

GLM‑5.2 activates roughly 40 billion parameters per token, requiring about 11 GB of shifting data; the remaining 17 billion parameters for attention, embeddings and shared experts stay resident in RAM.

The engine consists of a single C source file (c/glm.c, ~1,300 lines) and small headers; no external libraries, no GPU reliance.

Dense components are kept in RAM in int4 format, using about 9.9 GB; expert weights are stored on disk in a ~370 GB int4 container and loaded on demand via a per‑layer LRU cache and optional pinned hot‑store.

Quantization kernels support int8, packed int4, and packed int2 formats, employing AVX2 instructions for matrix multiplication; int8 matmuls achieve 1.4–2.5× speed over fp32, while int4 provides up to 1.8× speed in batch operations.

MTP speculative decoding uses an int8 draft head; acceptance rates range from 39 % to 59 % with 2.2–2.8 drafted tokens per forward pass, and the feature can be disabled until the cache warms.

The learning cache records expert usage and automatically pins the most frequently accessed experts in spare RAM, improving throughput as the session progresses.

Performance metrics: load time ~30 seconds; resident RAM ~9.9 GB; peak RSS during interactive use ~20 GB; cold decode cost ~11 GB of disk reads per token; throughput varies with storage speed, from ~0.05 tokens per second on a 1 GB/s NVMe VHDX to ~0.5–1 token per second on a PCIe 4 NVMe drive with 32 GB RAM, and up to ~2–4 tokens per second with faster storage and larger RAM allocations.

Requirements: Linux or WSL2, gcc with OpenMP and AVX2 support, at least 16 GB RAM, and a local NVMe drive with at least 370 GB free space for the int4 model; optional CUDA backend requires an NVIDIA driver and CUDA toolkit.

Limitations: random read performance of the storage device constrains cold token latency; sustained high read rates can generate swap activity and increase drive temperature; expert‑pinning and cache sizing are automatic but still limited by available RAM.

Community contributions are invited through issue reports of benchmark numbers, hardware donations, or code improvements; the project remains under an Apache 2.0 license for the engine and MIT for the model weights.

The engine’s design shows that large language models can be run on modest hardware through careful streaming and quantization, and performance is expected to increase as faster storage and larger memory become available.