Local inference with Qwen3.8-Flash-Next

Niko1221

@Niko1221 · Sep 24

Open-source inference engine with an OpenAI/Anthropic-style local API that fits the 125B MoE onto a 12GB consumer GPU plus system RAM. The project behind the model's local-running surge.

David Hendrickson

@TeksEdge · Sep 26

Tells the story of the 12GB RTX 5070 user who wrote Strata after finding llama.cpp too slow.

Alok

@analogalok · Aug 26

Posts a video of the 125B MoE running at about 21 tok/s decode with a quarter-million-token window on one gaming card, no quantised cache.

Unsloth AI

@UnslothAI · Aug 26

Launch-day announcement of GGUF quants and a guide for CPU and unified-memory setups.

JakeATX

@JakeATX · Sep 7

A llama.cpp variant optimised for Ampere cards, reporting very high token rates on 3090 and 3090 Ti, relevant for Flash-Next hosts.

azampatti

@azampatti · Sep 8

Quantisation and launch recipe focused purely on decode speed at 4-bit.

r0b0tlab

@r0b0tlab · Sep 17

Serving the model through ExLlamaV3 quantisation.

Lesilva

@Lesilva · Sep 29

Shrinks long-context memory use.

ucicelos

@ucicelos · Sep 1

Llama.cpp tree for hybrid integrated and external GPU machines.

zebgop-ops

@zebgop-ops · Aug 29

Root-cause forensics for a pipeline-parallel bug.

Investigated why Strata 0.1.40.1 left VRAM free on a 5060 Ti, 3090 and two 3060s running Flash-Next IQ3 at 262k context, and found it was by design. A useful look at how the engine sizes its caches.

Show-and-tell post for a fork with CPU decode optimisations.

A 64MB resource dashboard with rich logs that collects metrics across multiple inference engines on the network, better than the default API stats.

Walkthrough of a pruned build that claims about 98.7% of the coding score at much lower memory.

Shows that the KV cache of qwen4exp-style models can sit in system RAM, so a quant that barely fits in VRAM can still use the full context length.