MTP-RDMA inference cluster

Tensor-parallel 27B across two Macs over Thunderbolt RDMA — ~70 → 90–100 tok/s.

MLXThunderbolt RDMAPython

Dev-story

Challenge

We wanted a fast local LLM for our own coding agent (Pi) with no cloud API, but a 27B model on a single Mac decodes at only ~60–73 tok/s.

What we did

We built our own native multi-token-prediction (MTP) runtime and ran MLX tensor parallelism with a distributed pattern across two Macs over a Thunderbolt RDMA link: both ranks run the same generation loop while rank 0 serves an OpenAI-compatible API, and a Swift menu-bar app plus a control-plane daemon auto-discover the nodes and launch the pair at login.

Result

The pair decodes at ~85–100 tok/s (median ~95 fast / ~86 smart) versus ~60–73 solo, powering our Pi coding agent locally with fresh TTFT ~0.5 s.

Dev-story article

MTP-RDMA inference cluster: how the project was built

We build with our own coding agent, Pi, all day, and we did not want its brain to be a rented cloud API. But a 27B model on a single Mac decodes at only ~60-73 tokens per second: usable, not fast. The interesting question was whether two ordinary Macs, wired together, could behave like one bigger machine.

Sections

Modules

Stack

MLX + Thunderbolt RDMA

Why the project exists

We wanted a fast local LLM for our own coding agent (Pi) with no cloud API, but a 27B model on a single Mac decodes at only ~60–73 tok/s.

What was built

We joined two Macs over a Thunderbolt cable with RDMA and split the 27B model across both with MLX tensor parallelism. Both ranks run the exact same generation loop in lockstep, and rank 0 exposes an OpenAI-compatible API that Pi talks to. A Swift menu-bar app and a small control-plane daemon auto-discover the two nodes and launch the pair at login; if one Mac is answering solo, pressing Start Pair drains that solo server into the pair and hands control back safely when it stops.

Main modules and user path

M01

Pair versus solo was settled by measurement, not vibes: solo runs ~60-73 tok/s, the pair ~85-100 (median ~95 with thinking off, ~86 with reasoning on). The win is topology — the same model, split across two boxes — not a bigger model.

M02

The device matrix that decides which Thunderbolt interface talks to which is derived from the live interface at launch, so nothing about a specific Mac is hardcoded and a fresh machine reproduces the measured runtime.

M03

We tuned the served 27B itself through a long A/B: a smart-reduction pass cut reasoning tokens ~24-31% with no measurable quality loss, and a fast (thinking-off) and a smart (thinking-on) profile both serve from one loaded copy of the weights, so switching profile costs no reload.

M04

We deliberately did not rewrite the runtime in another language: we measured that the Python share of decode is only ~1-3%, and a license audit showed a rewrite gave zero legal benefit either. Two tempting dead-ends, ruled out with evidence instead of assumption.

M05

Thermals turned out to matter under sustained load, so fans are restored through a daemon socket rather than an app-killing CLI, and a patched compute overlay was A/B tested against stock and kept because it was ~10-14% faster.

Architecture and technology decisions

Built with MLX, Thunderbolt RDMA, Python.

A Swift menu-bar app plus a control-plane daemon (native Login Item, node auto-discovery, no SSH in production) drive MLX tensor parallelism over a Thunderbolt RDMA link; rank 0 serves an OpenAI-compatible API, serving is multimodal (text and image), and hermetic node self-tests plus post-hardening speed-parity replays act as regression guards under private Python and Swift CI.

Result and lessons

The pair decodes at ~85–100 tok/s (median ~95 fast / ~86 smart) versus ~60–73 solo, powering our Pi coding agent locally with fresh TTFT ~0.5 s.

A private, on-device 27B that decodes at ~95 tok/s and powers our Pi coding agent with fresh TTFT ~0.5s — no cloud, no per-token bill, and a pair that starts itself at login. The whole point held: two Macs on a desk, joined over one cable, feel like one faster machine.