Mesh LLM provides a distributed AI computing platform that aggregates GPU and memory resources across multiple devices into a single OpenAI‑compatible API. The system lets teams run larger models without purchasing additional hardware and avoids reliance on centralized providers. It can be deployed privately or publicly.
AI model serving typically requires access to a data center or a metered API, which imposes cost and control constraints. Many organizations hold GPUs in offices or closets but lack a method to coordinate them as a unified compute pool.
Mesh LLM runs an iroh endpoint on each node, which serves as a public key for authenticated QUIC connections. Nodes can execute models locally, forward requests to peers, or split a model across several machines using a pipeline mode that partitions layers. The split mode, internally called "Skippy," divides a model into stage ranges that are processed on different nodes, allowing modest machines to handle models that exceed the capacity of any single device. A manifest describes the capabilities of each plugin, and the runtime starts them, routes calls, and exposes their functions over MCP, HTTP, inference, and mesh events. The catalog includes more than 40 models ranging from half‑a‑billion‑parameter models that fit on a laptop to 235B mixture‑of‑experts giants.
The whole protocol uses three ALPN‑negotiated QUIC connections: the main mesh for gossip, routing, HTTP tunnels, and plugin channels; a control plane for configuration sync and ownership attestation; and a latency‑sensitive stage for split‑model activation transport. Within the main connection, each bidirectional stream is identified by a single leading byte that indicates its type, such as gossip, tunnel, route request, or peer‑down events.
The lightweight client, about 18 MB, presents itself as localhost:9337/v1 to any standard OpenAI client. Users can join a public mesh or configure a private deployment. A mobile app using iroh’s Swift SDK is planned, targeting the emerging ACP agent standard so other clients can join the mesh.
Mesh LLM seeks to replace closed server ecosystems with peer‑to‑peer infrastructure, though broader adoption will determine its impact.

