Qwen3.8-Flash-Next

Reports an IQ3_XXS run on a 24GB Radeon and slow DDR4 that beat llama.cpp's 22 tok/s on the same rig. Evidence that cheap RAM can be enough.

Hardware notes, vLLM work and benchmarks from an unusual inference machine built around cheap Ascend cards, which are not CUDA drop-ins.

Shows three Strata patches that let a 5090 plus a 4070 Ti Super on a PCIe x1 slot run the 76GB IQ3_XXS quant at 1,800 tok/s prompt processing.

Walkthrough of a pruned build that claims about 98.7% of the coding score at much lower memory.

A lab used a Dell B300 for two fine-tunes of Flash-Next, pruning experts from 512 to 288 per layer with REAP and retraining at NVFP4, GGUF included.

Benchmarked Flash-Next on a 12GB VRAM, 64GB RAM laptop and reached 50 tok/s decode and 1500 tok/s prefill, saying other llama.cpp forks were nowhere close.

Shares a metrics dashboard from agentic coding runs on 4x R9700 using an MXFP4 quant, saying speed and quality exceeded his expectations.

A team built an engine that keeps most of the NVFP4 model on SSD and reads experts as tokens need them, reaching 9-10 tok/s decode on a 16GB card with 32GB RAM.

Transferred the pretrained n-gram table into a tiny Qwen model and got 5% lower validation perplexity, trained mostly on free Kaggle GPUs. A resourceful research experiment.

Announced by the builder at UkisAI: a Qwen-based family trained to avoid pathological overthinking, with Flash-Next and 27B variants and 350k+ downloads in 13 days.

u/KnownAd4832

@KnownAd4832 · Sep 24

Went from 15 tok/s in llama.cpp to about 65 tok/s by writing an inference engine for this one model and this class of PC, with 2-bit RCO-GSQ quants running faster still.

Ran Intel AutoRound W4A16 Flash-Next on four V620 cards for about three hours with an OMP harness, letting it test and fix a game itself in two browsers.

u/peonist-ai

@peonist-ai · Sep 19

The Halogen engine author fixed slowdowns at deep context, raising decode at a million tokens from 27 to 38 tok/s on the same machine.

Shows that the KV cache of qwen4exp-style models can sit in system RAM, so a quant that barely fits in VRAM can still use the full context length.

Walks from a bare 6 tok/s up through successive patches on a middle-tier PC with Gen4 NVMe, documenting what actually helped. A guide others can follow.

Non-uniform quant repo that Strata users commonly pair with the engine.

Detailed benchmark session on a Strix Halo mini PC covering the n-gram table trick.

A co-creator of the MLX-serve engine support shows Flash-Next staying coherent at about 760k tokens of context with an 8-bit KV cache, demonstrated in a video.

Describes growing from an Arc B580 to a dual-R9700 box that runs both Qwen3.8 models well under vLLM. A practical value case for 32GB workstation Radeons over a pricey 5090.