Gaming gpu with Qwen3.8-Flash-Next

sergqwer

@sergqwer · Sep 29

Adds NVIDIA's NVFP4 quantisation to Strata so a single 12GB+ RTX 20 to 50-series card can host the MoE.

DominikBucko

@DominikBucko · Aug 30

Dual-3090 build with 128GB RAM reporting about 4,200 tok/s prefill and 111 tok/s decode on long prompts.

Apolog1ze-Dev

@Apolog1ze-Dev · Sep 5

A bespoke inference engine for the Qwen4-style architecture that runs Flash-Next on a 16GB card.

lifeidle

@lifeidle · Sep 10

IQ3_XXS 177B build served from a 24GB laptop GPU with 64GB RAM.

lna-lab

@lna-lab · Sep 1

Measured run using 6.6GiB of VRAM and about 48GiB of RAM on a laptop-class card.

dtm-beep

@dtm-beep · Sep 23

Build and run notes for the AtomicChat 4.27-bpw quant with multi-token prediction on a mid-range gaming PC.

xenodeve

@xenodeve · Sep 27

Runs the 177B build across RTX 5060 Ti and 4070 Super cards.

alesha-pro

@alesha-pro · Aug 28

262K context with vision and FP8 KV cache on four used GPUs.

halt95

@halt95 · Sep 2

Showcases long-context concurrent serving on used cards.

HaberstrohSystems

@HaberstrohSystems · Sep 3

SGLang patch and kernels for a single mid-range card.

ruashots

@ruashots · Aug 27

A homelab owner's practical write-up and configs.

loktar00

@loktar00 · Aug 29

Ampere-compatible reader plus benchmark receipts.

tonyd2wild

@tonyd2wild · Aug 26

Upstream vLLM run with full 262K context.

solarkyle

@solarkyle · Aug 27

Speculative-decoding variants benchmarked on one card.

Went from sceptic to convert after trying Strata on a low-end 5060 Ti with 32GB RAM and showed what the speed feels like in real demos.

u/fuzhongkai

@fuzhongkai · Oct 3

Ran the 176B MoE on an ordinary gaming laptop with 32GB RAM using his own open-source TensorSharp engine, to see how far a normal machine can go.

Shows three Strata patches that let a 5090 plus a 4070 Ti Super on a PCIe x1 slot run the 76GB IQ3_XXS quant at 1,800 tok/s prompt processing.

Benchmarked Flash-Next on a 12GB VRAM, 64GB RAM laptop and reached 50 tok/s decode and 1500 tok/s prefill, saying other llama.cpp forks were nowhere close.

A team built an engine that keeps most of the NVFP4 model on SSD and reads experts as tokens need them, reaching 9-10 tok/s decode on a 16GB card with 32GB RAM.

u/KnownAd4832

@KnownAd4832 · Sep 24

Went from 15 tok/s in llama.cpp to about 65 tok/s by writing an inference engine for this one model and this class of PC, with 2-bit RCO-GSQ quants running faster still.

Walks from a bare 6 tok/s up through successive patches on a middle-tier PC with Gen4 NVMe, documenting what actually helped. A guide others can follow.