exllamav3-amd: WMMA GEMV port gives 41 tok/s on Strix Halo

Ports ExLlamaV3 kernels to RDNA 3.5 with grouped MoE, int8 mixers and MTP. The author quantizes frontier models down to fit smaller GPUs.

Open on GitHub

Preview of exllamav3-amd: WMMA GEMV port gives 41 tok/s on Strix Halo

Launch: Qwen3.8-Flash-Next
Use case: Amd local
Format: Repo
Source: github.com

Victor Cruz

I quantize frontier models down into EXL3 and GGUF to fit on smaller GPUs, while also creating the most optimal recipes

See all

peonist-ai

@peonist-ai · Aug 26

Container-based server tuned for the gfx1151 Strix Halo APU, aimed at squeezing maximum tokens per second from the 125B model on a mini-PC.

IIIIIllllIIIIIlllll

@IIIIIllllIIIIIlllll · Sep 19

An inference engine written for Ryzen AI MAX 395 on Linux and Windows that deploys only Qwen-family MoE models. The author is an automation engineer, not a full-time programmer, who built it to get more out of his mini PC.

pwilkin

@pwilkin · Sep 5

Optimised AMD toolchain by a llama.cpp contributor.

Niko1221

@Niko1221 · Sep 24

Open-source inference engine with an OpenAI/Anthropic-style local API that fits the 125B MoE onto a 12GB consumer GPU plus system RAM. The project behind the model's local-running surge.

Open on GitHub

exllamav3-amd: WMMA GEMV port gives 41 tok/s on Strix Halo

More built with Qwen3.8-Flash-Next