Qwen3.8-Flash-Next W4A16-FP8PLE on 1x RTX 3090 — CPU cold experts, MTP3, 128K (DominikBucko)

community Community submission — verify on your own hardware.

recipe by DominikBucko↗, albucino (checkpoint)↗ · listed by localai-recipes

revision r2 · commit 088542fd7b · updated 10 d ago · license CC-BY-4.0

chat code mtp-speculative cpu-offload single-gpu long-context

Single-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in the albucino W4A16-FP8PLE build (@ ef55414, same checkpoint as the 2x3090 runtime): the GPU holds attention + dense weights + the 32 busiest experts per layer, a pinned AVX2 CPU thread pool computes the other 480 experts straight from RAM during decode (GPU takes ~22% over PCIe), and prefill streams all experts through the GPU in 8,192-token chunks. INT8 KV cache fits the full 135,168-token context in 3.05 GB; MTP 3-token draft; PLE table read in place from NVMe. The most common local-AI machine — one 24 GB card + 64 GB RAM. Author-measured (submitter T0, release image): 131,099-token prompt reads at 2,106 tok/s (62 s to first token) then decodes 47.4 tok/s; 43-51 tok/s decode range on 32 cores; 32K prompt at 2,141-2,219 tok/s prefill. Same vLLM runtime as the 2x3090 recipe, repackaged for half the hardware, nothing pruned. Argv transcribed verbatim from scripts/serve-container.sh @ 10b24c2.

Launch command

vllm serve /model --served-model-name Qwen3.8-Flash-Next --host 0.0.0.0 --port 8000 --tensor-parallel-size 1 --distributed-executor-backend mp --moe-backend humming --dtype bfloat16 --language-model-only --load-format safetensors --safetensors-load-strategy lazy --max-parallel-loading-workers 1 --max-model-len 135168 --max-num-seqs 1 --max-num-batched-tokens 8192 --kv-cache-dtype int8_per_token_head --kv-cache-memory-bytes 3050000000 --enable-chunked-prefill --enable-prefix-caching --mamba-cache-mode align --async-scheduling --disable-custom-all-reduce --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY"}' --trust-remote-code --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --speculative-config '{"method":"mtp","num_speculative_tokens":3,"use_local_argmax_reduction":true,"model":"/model/runtime/mtp-int4-g32"}'

runtime

env: 'CUDA_VISIBLE_DEVICES=0 | QWEN38_CPU_EXPERTS=1 QWEN38_CPU_EXPERTS_MODEL=/model | QWEN38_PLE_MMAP=1 QWEN38_PLE_MMAP_DIR=/model QWEN38_PLE_PREFAULT=0 | QWEN38_STREAM_STAGE=0 QWEN38_STAGE_OVERLAP=0 CPU_OFFLOAD_GB=0 | VLLM_PLE_CPU_OFFLOAD=1 VLLM_FORCE_DYNAMIC_SPEC_SCHEDULING=1 | VLLM_WNA16_DYNAMIC_LRU=0 VLLM_WNA16_MIXED_VMM_HOT_CACHE=0 VLLM_WNA16_STATIC_HOT_CACHE_SIZE=0 | VLLM_WNA16_STATIC_HOT_CACHE_MAX_TOKENS=16 VLLM_WNA16_STATIC_HOT_CACHE_FILE=<repo>/configs/static_hot_cache_rankings.json | MTP_DEPTH=3 | QWEN38_HOT_ONLY=32 QWEN38_EXPERT_ARENA_SLOTS=auto QWEN38_STREAM_GROUP=80 QWEN38_CPU_PREFILL_MAX=384 QWEN38_EXPERT_PROMOTE=8 QWEN38_GPU_SHARE_FRAC=0.22 | VLLM_PREFIX_CACHE_RETENTION_INTERVAL=3200 (must be a multiple of 3200 with int8 KV) | QWEN38_ASYNC_SCHEDULING=1 QWEN38_TRITON_SKINNY=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | QWEN38_CPU_EXPERTS_CPUS=auto QWEN38_MAIN_CPUS=auto (resolved by vllm/qwen38_host.py) (verbatim from scripts/serve-container.sh @ 10b24c2 with configs/3090-64gb-128k.env defaults inlined)'
notes: 'Run path: git clone the repo @ 10b24c2, hf download albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE --revision ef554143369a706525336f6b42a09094835dc077 --local-dir <MODEL_DIR> (~129 GB on NVMe — read while serving), cp .env.example .env, set MODEL_DIR, then IMAGE=ghcr.io/dominikbucko/qwen38-flash-next-3090@sha256:7f176605b59c462af21b1b62fd854f9f3cca96e3d11a295581297483d45490a3 make serve (v0.1.0 image, or make build-image first). The container mounts MODEL_DIR at /model and execs scripts/serve-container.sh, the verbatim argv source above; make serve runs scripts/preflight.sh first (GPU/RAM/CPU/driver/disk checks). Container gets installed RAM minus 8 GiB (MEMORY_LIMIT=auto) and sizes its expert arena to fit (56 GiB serving cap on the 64 GB benchmark host). Host needs: one 24 GB GPU (RTX 3090 tested; 3090 Ti/4090 should work), 64 GB RAM (96 GB+ keeps every expert resident), x86-64 AVX2 CPU, NVMe ~130 GB free, NVIDIA driver 580+. GPU peaks 23.4 GB; INT8 KV holds 141,504 tokens in 3.05 GB. One request at a time, up to 135,168 tokens. Decode scales with RAM bandwidth (43-51 tok/s @32 cores, 39-47 @16, 34-36 @8). Known issue: occasional slower prefill path ~5.5 s per 8K chunk (open upstream). Verify with scripts/smoke_test.py / make bench.'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts.json

Engine

vllm · 8e685d19↗ · ghcr.io/dominikbucko/qwen38-flash-next-3090@sha256:7f176605b59c462af21b1b62fd854f9f3cca96e3d11a295581297483d45490a3↗

Model

Qwen3.8-Flash-Next-W4A16-FP8PLE (w4a16-fp8-ple) · albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE↗
AA intelligence 40 (Artificial Analysis · measured)

Hardware

NVIDIA RTX 3090 24GB ×1 (Ampere)

Context

max context: 132K tokens

Performance

Speed by context

Context	Conc.	Prefill	Decode	TTFT
32,768	1	2141 tok/s	50.3 tok/s	15300 ms	reported
131,072	1	2106 tok/s	47.4 tok/s	62200 ms	reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notes	Author README results tables (repo @ 10b24c2, RTX 3090 + 64 GB RAM, release image v0.1.0, one request at a time, greedy with MTP, best of 2 runs, submitter T0 — no run file ingested; import via POST /api/v1/benchmarks to upgrade). 131,099+512: first token 62.2 s, 2,106 tok/s prefill, 47.4 tok/s decode (second run 90.3 s / 1,451 / 43.3). 32,799+512: 15.3 s, 2,141 tok/s, 50.3 tok/s. 8,218+1,024: 5.2 s, 1,570 tok/s, 49.1 tok/s. CPU core scaling (Zen 3/DDR4 proxy): 32 cores 43-51 tok/s decode, 16 cores 39-47, 8 cores 34-36. GPU memory peaked 23.4 GB, no preemptions.

notes

Author README results tables (repo @ 10b24c2, RTX 3090 + 64 GB RAM, release image v0.1.0, one request at a time, greedy with MTP, best of 2 runs, submitter T0 — no run file ingested; import via POST /api/v1/benchmarks to upgrade). 131,099+512: first token 62.2 s, 2,106 tok/s prefill, 47.4 tok/s decode (second run 90.3 s / 1,451 / 43.3). 32,799+512: 15.3 s, 2,141 tok/s, 50.3 tok/s. 8,218+1,024: 5.2 s, 1,570 tok/s, 49.1 tok/s. CPU core scaling (Zen 3/DDR4 proxy): 32 cores 43-51 tok/s decode, 16 cores 39-47, 8 cores 34-36. GPU memory peaked 23.4 GB, no preemptions.

Install requirements

Docker (or a compatible OCI container runtime)
NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container

NVIDIA driver floor: unknown - check the image tag

Source: NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03).

Map the placeholders to your rig

Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.
GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.
Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.
Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged.

Downloads

Weights https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE
ef554143369a706525336f6b42a09094835dc077 — same checkpoint as the 2x RTX 3090 runtime: Intel AutoRound W4A16 target, FP8 PLE table, INT4 MTP draft (repro.lock)
Engine image https://ghcr.io/dominikbucko/qwen38-flash-next-3090
sha256:7f176605b59c462af21b1b62fd854f9f3cca96e3d11a295581297483d45490a3 — v0.1.0 release image (tags v0.1.0, sha-e4c7d3d); base runtime = the 2x3090 image v0.4.0 (sha256:139e796c)
Reference https://github.com/DominikBucko/qwen38-flash-next-3090
10b24c282f59348037096a2259d3ba0948f1b202 — source recipe repo @ HEAD (scripts/serve-container.sh is the verbatim argv source; configs/3090-64gb-128k.env, repro.lock.json all at this commit)

Revision notes

Community ingest of DominikBucko/qwen38-flash-next-3090 @ 10b24c282f59348037096a2259d3ba0948f1b202 (owner-named by Casey Oct 6; last push 2026-09-30, Apache-2.0). Launch argv transcribed verbatim from scripts/serve-container.sh @ 10b24c2 with configs/3090-64gb-128k.env defaults inlined (the default profile); env exports verbatim from the same script. Token-level check done against the raw script before POST. Perf numbers are the author's README tables, logged claimed_by=submitter T0. Same checkpoint and vLLM runtime as the sibling 2x3090 recipe (this wave's recipe A) — repackaged for one card with CPU cold experts.

Revision history

4075742544 recipe: update qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts (r2: /max_context_len, /storage_requirements) 10 d ago diff
87bdcc4efa recipe: create qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts 24 d ago

Fork

Export

recipe.yaml (raw)