{"slug":"qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts","title":"Qwen3.8-Flash-Next W4A16-FP8PLE on 1x RTX 3090 \u2014 CPU cold experts, MTP3, 128K (DominikBucko)","summary":"Single-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in the albucino W4A16-FP8PLE build (@ ef55414, same checkpoint as the 2x3090 runtime): the GPU holds attention + dense weights + the 32 busiest experts per layer, a pinned AVX2 CPU thread pool computes the other 480 experts straight from RAM during decode (GPU takes ~22% over PCIe), and prefill streams all experts through the GPU in 8,192-token chunks. INT8 KV cache fits the full 135,168-token context in 3.05 GB; MTP 3-token draft; PLE table read in place from NVMe. The most common local-AI machine \u2014 one 24 GB card + 64 GB RAM. Author-measured (submitter T0, release image): 131,099-token prompt reads at 2,106 tok/s (62 s to first token) then decodes 47.4 tok/s; 43-51 tok/s decode range on 32 cores; 32K prompt at 2,141-2,219 tok/s prefill. Same vLLM runtime as the 2x3090 recipe, repackaged for half the hardware, nothing pruned. Argv transcribed verbatim from scripts/serve-container.sh @ 10b24c2.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"088542fd7b452ae34d7f6b0253b939a3ef6b350d","content_hash":"a354418ea2f8a6dbcef09d963e5901758b7dd549","created_at":"2026-10-20 15:34:09.15642-06","notes":"Community ingest of DominikBucko/qwen38-flash-next-3090 @ 10b24c282f59348037096a2259d3ba0948f1b202 (owner-named by Casey Oct 6; last push 2026-09-30, Apache-2.0). Launch argv transcribed verbatim from scripts/serve-container.sh @ 10b24c2 with configs/3090-64gb-128k.env defaults inlined (the default profile); env exports verbatim from the same script. Token-level check done against the raw script before POST. Perf numbers are the author's README tables, logged claimed_by=submitter T0. Same checkpoint and vLLM runtime as the sibling 2x3090 recipe (this wave's recipe A) \u2014 repackaged for one card with CPU cold experts."},"engine_source":{"type":"docker_image","git_url":null,"base_engine":"vllm","pip_package":null,"base_version":"0.1.dev20073+g8e685d198 (qwen38-flash-next runtime)","image_digest":"ghcr.io/dominikbucko/qwen38-flash-next-3090@sha256:7f176605b59c462af21b1b62fd854f9f3cca96e3d11a295581297483d45490a3","install_note":null,"pinned_commit":"8e685d198"},"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/qwen38-flash-next-3090"},{"name":"albucino (checkpoint)","url":"https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/model","--served-model-name","Qwen3.8-Flash-Next","--host","0.0.0.0","--port","8000","--tensor-parallel-size","1","--distributed-executor-backend","mp","--moe-backend","humming","--dtype","bfloat16","--language-model-only","--load-format","safetensors","--safetensors-load-strategy","lazy","--max-parallel-loading-workers","1","--max-model-len","135168","--max-num-seqs","1","--max-num-batched-tokens","8192","--kv-cache-dtype","int8_per_token_head","--kv-cache-memory-bytes","3050000000","--enable-chunked-prefill","--enable-prefix-caching","--mamba-cache-mode","align","--async-scheduling","--disable-custom-all-reduce","--compilation-config","{\"mode\":0,\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}","--trust-remote-code","--enable-auto-tool-choice","--tool-call-parser","qwen3_coder","--reasoning-parser","qwen3","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":3,\"use_local_argmax_reduction\":true,\"model\":\"/model/runtime/mtp-int4-g32\"}"],"runtime":{"env":"CUDA_VISIBLE_DEVICES=0 | QWEN38_CPU_EXPERTS=1 QWEN38_CPU_EXPERTS_MODEL=/model | QWEN38_PLE_MMAP=1 QWEN38_PLE_MMAP_DIR=/model QWEN38_PLE_PREFAULT=0 | QWEN38_STREAM_STAGE=0 QWEN38_STAGE_OVERLAP=0 CPU_OFFLOAD_GB=0 | VLLM_PLE_CPU_OFFLOAD=1 VLLM_FORCE_DYNAMIC_SPEC_SCHEDULING=1 | VLLM_WNA16_DYNAMIC_LRU=0 VLLM_WNA16_MIXED_VMM_HOT_CACHE=0 VLLM_WNA16_STATIC_HOT_CACHE_SIZE=0 | VLLM_WNA16_STATIC_HOT_CACHE_MAX_TOKENS=16 VLLM_WNA16_STATIC_HOT_CACHE_FILE=<repo>/configs/static_hot_cache_rankings.json | MTP_DEPTH=3 | QWEN38_HOT_ONLY=32 QWEN38_EXPERT_ARENA_SLOTS=auto QWEN38_STREAM_GROUP=80 QWEN38_CPU_PREFILL_MAX=384 QWEN38_EXPERT_PROMOTE=8 QWEN38_GPU_SHARE_FRAC=0.22 | VLLM_PREFIX_CACHE_RETENTION_INTERVAL=3200 (must be a multiple of 3200 with int8 KV) | QWEN38_ASYNC_SCHEDULING=1 QWEN38_TRITON_SKINNY=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | QWEN38_CPU_EXPERTS_CPUS=auto QWEN38_MAIN_CPUS=auto (resolved by vllm/qwen38_host.py) (verbatim from scripts/serve-container.sh @ 10b24c2 with configs/3090-64gb-128k.env defaults inlined)","notes":"Run path: git clone the repo @ 10b24c2, hf download albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE --revision ef554143369a706525336f6b42a09094835dc077 --local-dir <MODEL_DIR> (~129 GB on NVMe \u2014 read while serving), cp .env.example .env, set MODEL_DIR, then IMAGE=ghcr.io/dominikbucko/qwen38-flash-next-3090@sha256:7f176605b59c462af21b1b62fd854f9f3cca96e3d11a295581297483d45490a3 make serve (v0.1.0 image, or make build-image first). The container mounts MODEL_DIR at /model and execs scripts/serve-container.sh, the verbatim argv source above; make serve runs scripts/preflight.sh first (GPU/RAM/CPU/driver/disk checks). Container gets installed RAM minus 8 GiB (MEMORY_LIMIT=auto) and sizes its expert arena to fit (56 GiB serving cap on the 64 GB benchmark host). Host needs: one 24 GB GPU (RTX 3090 tested; 3090 Ti/4090 should work), 64 GB RAM (96 GB+ keeps every expert resident), x86-64 AVX2 CPU, NVMe ~130 GB free, NVIDIA driver 580+. GPU peaks 23.4 GB; INT8 KV holds 141,504 tokens in 3.05 GB. One request at a time, up to 135,168 tokens. Decode scales with RAM bandwidth (43-51 tok/s @32 cores, 39-47 @16, 34-36 @8). Known issue: occasional slower prefill path ~5.5 s per 8K chunk (open upstream). Verify with scripts/smoke_test.py / make bench."},"max_context_len":135168,"storage_requirements":{"ram_gb":64}},"models":[{"slug":"qwen3-8-flash-next-w4a16-fp8ple","name":"Qwen3.8-Flash-Next-W4A16-FP8PLE","hf_repo":"albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE","quant":"w4a16-fp8-ple"}],"hardware":[{"slug":"rtx-3090","vendor":"nvidia","card":"RTX 3090","card_family":"rtx-3090","vram_gb":24,"device_count":1}],"generation":"ampere","tags":["cpu-offload","long-context","mtp-speculative","single-gpu"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author README results tables (repo @ 10b24c2, RTX 3090 + 64 GB RAM, release image v0.1.0, one request at a time, greedy with MTP, best of 2 runs, submitter T0 \u2014 no run file ingested; import via POST /api/v1/benchmarks to upgrade). 131,099+512: first token 62.2 s, 2,106 tok/s prefill, 47.4 tok/s decode (second run 90.3 s / 1,451 / 43.3). 32,799+512: 15.3 s, 2,141 tok/s, 50.3 tok/s. 8,218+1,024: 5.2 s, 1,570 tok/s, 49.1 tok/s. CPU core scaling (Zen 3/DDR4 proxy): 32 cores 43-51 tok/s decode, 16 cores 39-47, 8 cores 34-36. GPU memory peaked 23.4 GB, no preemptions.\", \"tok_s\": 47.39999999999999857891452847979962825775146484375, \"points\": [{\"notes\": \"131,099+512 best of 2 (README): 62.2 s TTFT, 2,106 tok/s prefill (second run 1,451), 47.4 tok/s decode (43.3 second run)\", \"ttft_ms\": 62200, \"concurrency\": 1, \"context_len\": 131072, \"decode_tok_s\": 47.39999999999999857891452847979962825775146484375, \"prefill_tok_s\": 2106}, {\"notes\": \"32,799+512 (README): 15.3 s TTFT, 2,141 tok/s prefill, 50.3 tok/s decode\", \"ttft_ms\": 15300, \"concurrency\": 1, \"context_len\": 32768, \"decode_tok_s\": 50.2999999999999971578290569595992565155029296875, \"prefill_tok_s\": 2141}], \"context_len\": 131099}"}],"perf_points":[{"id":1035,"context_len":32768,"prefill_tok_s":2141,"decode_tok_s":50.2999999999999971578290569595992565155029296875,"ttft_ms":15300,"concurrency":1,"claimed_by":"submitter","notes":"32,799+512 (README): 15.3 s TTFT, 2,141 tok/s prefill, 50.3 tok/s decode"},{"id":1034,"context_len":131072,"prefill_tok_s":2106,"decode_tok_s":47.39999999999999857891452847979962825775146484375,"ttft_ms":62200,"concurrency":1,"claimed_by":"submitter","notes":"131,099+512 best of 2 (README): 62.2 s TTFT, 2,106 tok/s prefill (second run 1,451), 47.4 tok/s decode (43.3 second run)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE","pin":"ef554143369a706525336f6b42a09094835dc077 \u2014 same checkpoint as the 2x RTX 3090 runtime: Intel AutoRound W4A16 target, FP8 PLE table, INT4 MTP draft (repro.lock)"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://ghcr.io/dominikbucko/qwen38-flash-next-3090","pin":"sha256:7f176605b59c462af21b1b62fd854f9f3cca96e3d11a295581297483d45490a3 \u2014 v0.1.0 release image (tags v0.1.0, sha-e4c7d3d); base runtime = the 2x3090 image v0.4.0 (sha256:139e796c)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/DominikBucko/qwen38-flash-next-3090","pin":"10b24c282f59348037096a2259d3ba0948f1b202 \u2014 source recipe repo @ HEAD (scripts/serve-container.sh is the verbatim argv source; configs/3090-64gb-128k.env, repro.lock.json all at this commit)"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts.json","alternate":"/r/qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:09.15642-06","created_at":"2026-10-06 17:12:38.509203-06","speeds":{"decode_tok_s":48.89999999999999857891452847979962825775146484375,"prefill_tok_s":2123.5,"leaderboard":"/api/v1/leaderboard.json"}}