Best model and runtime on a 128GB M5 Mac, tested on his own coding and agent tasks

Streams expert weights from SSD so a Mac that cannot hold the model in memory still runs it at roughly 12 tok/s. A neat trick for memory-poor Apple machines.

Open-source inference engine with an OpenAI/Anthropic-style local API that fits the 125B MoE onto a 12GB consumer GPU plus system RAM. The project behind the model's local-running surge.

Step-by-step recipe to squeeze the 125B model onto one DGX Spark instead of two, with config and measured throughput.

Container-based server tuned for the gfx1151 Strix Halo APU, aimed at squeezing maximum tokens per second from the 125B model on a mini-PC.