Offload most of Flash-Next's KV cache to RAM with little decode slowdown

sadnessdevilSep 16

Shows that the KV cache of qwen4exp-style models can sit in system RAM, so a quant that barely fits in VRAM can still use the full context length.

Open on Reddit

Launch: Qwen3.8-Flash-Next
Use case: Local inference
Format: Build
Source: reddit.com

sadnessdevil

See all

Niko1221

@Niko1221 · Sep 24

Open-source inference engine with an OpenAI/Anthropic-style local API that fits the 125B MoE onto a 12GB consumer GPU plus system RAM. The project behind the model's local-running surge.

David Hendrickson

@TeksEdge · Sep 26

Tells the story of the 12GB RTX 5070 user who wrote Strata after finding llama.cpp too slow.

Alok

@analogalok · Aug 26

Posts a video of the 125B MoE running at about 21 tok/s decode with a quarter-million-token window on one gaming card, no quantised cache.

Unsloth AI

@UnslothAI · Aug 26

Launch-day announcement of GGUF quants and a guide for CPU and unified-memory setups.

Open on Reddit

Offload most of Flash-Next's KV cache to RAM with little decode slowdown

More built with Qwen3.8-Flash-Next