Qwen3.8-Flash-Next

A homelab owner's practical write-up and configs.

TP2 with expert parallelism and MTP speculation.

MeldLabs' release that runs Flash-Next on Apple silicon laptops at about 45 tokens per second, with a demo video.

Ampere-compatible reader plus benchmark receipts.

Compresses the embedding table so a single Spark holds the whole thing at long context.

Hybrid MoE serving on a 128GB Spark.

Ports ExLlamaV3 kernels to RDNA 3.5 with grouped MoE, int8 mixers and MTP. The author quantizes frontier models down to fit smaller GPUs.

Draft-head speculation and carefully measured Vulkan settings that lift Flash-Next speed on Strix Halo. Notable for publishing the measurement method as well as the numbers.

A step-by-step serving recipe for running the 125B MoE on one Ryzen AI MAX+ 395 box, written by a hobbyist. Useful as a replicable baseline for that hardware.

CUDA-graph build for the workstation Blackwell.

Llama.cpp tree for hybrid integrated and external GPU machines.

Llama.cpp route with long context.

Build recipe, Lemonade deployment, an admission shim and benchmarks for Windows users, a platform most community recipes skip.

Upstream vLLM run with full 262K context.

Guide to running the EXL3 quant as a local Hermes Agent with ExLlamaV3 and native OpenAI tool calling on consumer GPUs.

ROCm patches speeding Flash-Next decode on AMD AI Pro cards.

Screen recording of the model reading a CLI guide and completing a task that the 27B failed.

SGLang NEXTN setup at about 37 tok/s on code.

FP8 and NVFP4 with MTP speculation on a dual-Blackwell box.

Speculative-decoding variants benchmarked on one card.

Reaches 1M KV tokens using FP8 PLE on a 91GB host, a cheap high-throughput setup.

Root-cause forensics for a pipeline-parallel bug.

A benchmark comparing GGUF quantization levels on generated code, helping choose the smallest quant that still codes well.

Tests Qwen3.8-27B and Flash-Next on two 3090s and finds a terse prompting style dramatically improves math scores.