A developer identified as SlvDev demonstrated running a small language model on an ESP32 microcontroller that costs about $10, achieving roughly 10 tokens per second, according to a GitHub repository and a Better Stack YouTube video.

Large language models typically require significant memory and processing power, often running on GPUs, and models such as TinyStories with 28.9 million parameters still demand more resources than the ESP32‑S3 can provide.

To make the model fit, the developer applied quantization, reducing weight precision from 16‑bit to 8‑bit and cutting memory requirements by about 75 percent, and used per‑layer‑embedding offloading to store most parameters in flash memory while keeping only the output head, embeddings, and KV cache in the chip’s PSRAM.

These techniques reduced the model’s memory footprint to approximately 2 MB and allowed inference at 9.88 tokens per second, which exceeds the average reading speed of a human.

While the demonstration proves that LLMs can operate on low‑power hardware, the model can generate short stories but is not suited for chatbots, code generation, or agent tasks; another model called Barista provides limited question answering on espresso topics at roughly double the speed.

Larger devices such as Raspberry Pi or smartphones can run bigger models like Google’s Gemma 4‑E2B‑it, which uses similar quantization and offloading methods to fit a 5.1‑billion‑parameter model within a gigabyte of memory.

The experiment highlights the feasibility of running LLMs on constrained devices, though practical applications remain limited and further optimization may be needed for broader use.