AirLLM, an open‑source software package, enables 70‑billion‑parameter large language models to run on a single 4‑GB GPU without quantization, distillation, or pruning.

The library decomposes models into layers and streams a single expert at a time, reducing VRAM requirements to the size of the active layer.

During inference, the model is split layer‑wise and stored on disk; only the currently needed layer resides in GPU memory.

Model compression, based on block‑wise weight quantization, can accelerate inference up to threefold with minimal accuracy loss, according to the developers.

Users must install bitsandbytes, a recent version of AirLLM (2.0 or later), and a CUDA 12‑compatible build of PyTorch; flash‑attention support is mandatory for K3 models.

The project released a K3 model support update in July 2026 requiring pip install of compressed‑tensors and flash‑attention, a CUDA 12 build of torch, and transformers 4.56.x. Earlier updates added 8‑bit and 4‑bit quantization support in August 2024, CPU inference in August 2024, and MacOS compatibility in December 2023.

The library works with a range of open models, including Qwen3‑235B, DeepSeek‑V3, Llama 3‑70B, and others, with VRAM needs ranging from about 1 GB for 8‑billion‑parameter models to 12 GB for 671‑billion‑parameter models.

Configuration options allow users to enable 4‑bit or 8‑bit compression, activate profiling mode, specify shard saving paths, provide a Hugging Face token for gated models, and delete original files to save disk space.

The source code remains publicly available on GitHub, and the developer invites contributions and feedback.