{"slug":"qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f","title":"Qwen3.8-Flash-Next W4A16-Merlin on 4x RTX 3090 \u2014 TP2xPP2+EP, MTP3, FP8 KV, 925K pool (halt95)","summary":"Four-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in halt95's W4A16-Merlin build (Intel AutoRound INT4 + FP8 PLE + MTP draft, @ 17284bf): TP2 x PP2 with expert parallel, manual layer partition 25,23, FP8 KV with a 924,993-token pool (three full 262K sessions resident; block-size 4096, prefix-match-unit 64), MTP 3-token probabilistic draft with RecoverSSM, host-mapped PLE pull transport, host-resident embeddings, side-cache layout + Model Runner V2, 42 images/request. Agent-focused: 92.5% of follow-up prompt tokens from cache across 8 sessions, median follow-up first token 1.61 s (0.96 s solo). Maintainer-measured (submitter T0, v2.5.1 bench card 2026-10-06): 5,219-5,533 tok/s cold prefill 10K-261K, 158-174 tok/s single-stream decode (thinking on), 534 tok/s aggregate at 8 concurrent, 262K needle exact, GSM8K 198/200. Engine is a SOURCE BUILD: vLLM v0.30.0 + 97 commits (release bundle verified by commit AND tree hash); argv verbatim from scripts/serve-v2.5.sh @ 28cbb72.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"d2343f0bc9d171b68356492709d3915f2e0eb610","content_hash":"8aefc67a1e986e515743a30617a02bbc169edc71","created_at":"2026-10-29 21:32:28.596454-06","notes":"Community ingest of halt95/qwen38-flash-next-3090s @ 28cbb7293f134c244b443fb43bbf6e74ff4d8e24, release v2.5.1 (owner-named by Casey Oct 6; repo pushed 2026-10-06; Apache-2.0 + Qwen Community 1.0 for weights). Launch argv transcribed verbatim from scripts/serve-v2.5.sh @ 28cbb72 (selector-off legacy path, token-level checked against the raw script before POST); env exports verbatim from the same script. Perf numbers are the maintainer's 2026-10-06 bench card, logged claimed_by=submitter T0. Credits per the repo's own Credits section: Qwen (base), Intel AutoRound + RadixArk (quant components), DominikBucko (MTP packing), peakcrosser7 Flash-Next branch behind vLLM PRs #53896/#53899, aikitoria open-kernel-module P2P patch. Sibling repo halt95/Swift1.5-qwen38-flash-next-3090s is a post-train variant \u2014 NOT in scope this wave."},"engine_source":{"type":"fork","git_url":"https://github.com/halt95/merlin-engine","base_engine":"vllm","pip_package":null,"base_version":"v0.30.0 + 97 commits (Flash-Next v2.5.1 build)","image_digest":null,"install_note":"Source build required \u2014 the vLLM fork ships as a release bundle, not an image (container image exists but is NOT rebuilt/GPU-served on v2.5.1 per README; the bare-metal route is the verified one). Build: clone the recipe repo @ 28cbb72, download release/v2.5.1/v2.5.1-from-upstream-v0.30.0.bundle.gz + build-artifacts-sm86-py313-cu130-v2.2.0.tar.gz from the repo's releases, then BUNDLE=<bundle> ARTIFACTS=<tarball> scripts/build-v2.5.sh ./vllm-v2.5 ./venv-v2.5 \u2014 the script verifies the base commit (vLLM v0.30.0 ced6857a), the fork commit 96a0d13b AND its tree hash 0df6d7ef, and hash-checks every build product before it serves. Needs python 3.13, CUDA 13.0+ driver (580+), ninja/C compiler for the first-serve FlashInfer/Triton JIT; compiled ops sm_86.","pinned_commit":"96a0d13b6e98ad4ef86e66a3f6ddb5dd9199a591"},"credit":[{"name":"halt95","url":"https://github.com/halt95/merlin-engine"},{"name":"Qwen (Qwen3.8-Flash-Next)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"Intel AutoRound (INT4 experts)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"RadixArk (FP8 PLE table)","url":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4"},{"name":"DominikBucko (MTP INT4 packing recipe)","url":"https://github.com/DominikBucko/qwen38-flash-next-2x3090"},{"name":"peakcrosser7 (Flash-Next vLLM branch)","url":"https://github.com/peakcrosser7/vllm/commits/release/qwen38next_offload"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["./venv-v2.5/bin/vllm","serve","~/models/Flash-Next-Merlin","--served-model-name","flash-next-v2","flash-next","flash-mtp","flash-next-mtp","--host","127.0.0.1","--port","8000","--tensor-parallel-size","2","--pipeline-parallel-size","2","--enable-expert-parallel","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":3,\"draft_sample_method\":\"probabilistic\"}","--max-model-len","262144","--max-num-seqs","8","--max-num-batched-tokens","1024","--gpu-memory-utilization","0.96","--kv-cache-memory","4100000000","--kv-cache-dtype","fp8_e4m3","--limit-mm-per-prompt","{\"image\":42,\"video\":0}","--mm-processor-cache-gb","0","--mm-processor-kwargs","{\"size\":{\"longest_edge\":4194304,\"shortest_edge\":65536}}","--compilation-config","{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[1,2,3,4,6,8,12,16,24,32]}","--enable-prefix-caching","--enable-prompt-tokens-details","--no-enable-flashinfer-autotune","--enable-auto-tool-choice","--tool-call-parser","qwen3_coder","--reasoning-parser","qwen3","--override-generation-config","{\"temperature\":1.0,\"top_p\":0.95,\"top_k\":20,\"min_p\":0.0,\"presence_penalty\":0.0,\"repetition_penalty\":1.0}","--default-chat-template-kwargs","{\"enable_thinking\": true, \"reasoning_effort\": \"low\"}","--shutdown-timeout","60","--block-size","4096","--use-replayssm","--prefix-match-unit","64"],"runtime":{"env":"TREE=./vllm-v2.5 VENV=./venv-v2.5 | CUDA_VISIBLE_DEVICES=0,1,2,3 | CUDA_DEVICE_ORDER=PCI_BUS_ID | VLLM_SKIP_P2P_CHECK=1 NCCL_P2P_LEVEL=SYS NCCL_PROTO=LL | VLLM_PP_LAYER_PARTITION=25,23 | VLLM_E2_PLE_PULL_TRANSPORT=1 VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 | VLLM_PLE_OFFLOAD_HOME_DEVICE=0 | VLLM_HOST_EMBED_TABLE=1 | VLLM_USE_BREAKABLE_CUDAGRAPH=0 | VLLM_QSA_KV_SCALES=<repo>/scales/qsa_kv_scales_262k.json (required \u2014 without it FP8 KV runs at scale 1.0) | VLLM_CACHE_ROOT=<repo>/.vllm-cache-v2.5 | HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 | MERLIN_FULL_K=1 MERLIN_QSA_SORT=1 MERLIN_TIE_DET=1 MERLIN_TIE_RECENT=1 MERLIN_TOPK_SORTED_EMIT=1 | VLLM_E2_GUARD_MODE=warn | VLLM_E2_SIDE_CACHE_LAYOUT=1 VLLM_USE_V2_MODEL_RUNNER=1 | VLLM_E2_MAMBA_RETAIN_CHECKPOINT=1 | PYTHONPATH=<vllm tree> VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 (all verbatim from scripts/serve-v2.5.sh @ 28cbb72, selector-off legacy path). Opt-in: AUTO_TOPO=1 topology selector; COUNTERS=1 logging-only counters; HOST=0.0.0.0 then set VLLM_API_KEY.","notes":"Run path: git clone --branch v2.5.1 (repo @ 28cbb72), fetch the v2.5.1 source bundle + the unchanged-since-v2.2.0 compiled-ops tarball from the repo's releases, hf download halt95/Qwen3.8-Flash-Next-W4A16-Merlin --local-dir ~/models/Flash-Next-Merlin (115 GiB), then TREE=./vllm-v2.5 VENV=./venv-v2.5 CACHE_ROOT=./.vllm-cache-v2.5 scripts/serve-v2.5.sh ~/models/Flash-Next-Merlin \u2014 the script execs vllm serve with the argv above. Startup ~5-6 min first boot (kernel JIT), 2-3 min after; ready when the log shows 'Application startup complete' and 'GPU KV cache size: 924,993 tokens'. Docker compose variant exists (MODEL_DIR=... docker compose up -d) but the image is not rebuilt/GPU-served on v2.5.1. Requirements: 4x RTX 3090 (sm_86) headless, P2P recommended (runs without at slower prefill), no NVLink; 96 GB host RAM (n-gram table ~48 GiB + ~4.2 GiB pinned embeddings; measured floor ~69 GiB), /dev/shm >= 1 GB, ~250 GB disk. The checkpoint config needs ple_embedding_dtype: float8_e4m3fn inside text_config (present in HF since 2026-09-18). Container needs shm_size 8g + memlock -1 + stop_grace_period 70s. Proxmox LXC recipes in lxc/."},"kv_pool_tokens":924993,"max_context_len":262144,"storage_requirements":{"ram_gb":96}},"models":[{"slug":"qwen3-8-flash-next-w4a16-merlin","name":"Qwen3.8-Flash-Next-W4A16-Merlin","hf_repo":"halt95/Qwen3.8-Flash-Next-W4A16-Merlin","quant":"w4a16-fp8-kv"}],"hardware":[{"slug":"rtx-3090-x4","vendor":"nvidia","card":"RTX 3090","card_family":"rtx-3090","vram_gb":24,"device_count":4}],"generation":"ampere","tags":["agent-cache","fp8-kv","long-context","mtp-speculative","pipeline-parallel"],"tasks":["chat","code","vision"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Maintainer bench card benchmarks/2026-10-06/BENCH-CARD.md (repo @ 28cbb72, reference host 4x RTX 3090 220 W PCIe P2P driver 610.43.02, submitter T0 \u2014 no run file ingested; import via POST /api/v1/benchmarks to upgrade). Single-stream decode (thinking on, MTP): 157.9 tok/s @4K, 159.2 @32K, 161.8 @131K, 174.3 @261K (thinking off ~115-118). Aggregate 533.7 tok/s at N=8, 124.4 at N=1. Cold prefill (ctx_pp medians): 5,219 tok/s @10K to 5,236 @261K (peak 5,533 @50K). Follow-up cache reuse 92.5% @8 sessions (v2.2.0: 7.5%), median follow-up TTFT 1.61 s (0.96 s one conversation). KV pool 924,993 FP8 tokens; 3x full 262K or 6x 131K sessions resident, 0 preemptions. Quality vs BF16 teacher: divergence 0.0334, 262,144-token needle exact, GSM8K 198/200, structured output 80/80.\", \"tok_s\": 533.700000000000045474735088646411895751953125, \"points\": [{\"notes\": \"bench card 2026-10-06 @ repo 28cbb72: ctx_pp 5,484 tok/s median @100K depth; ctx_tg 161.8 tok/s @131K thinking-on (ITL 19.85 ms, ~3.13 tokens/event with MTP)\", \"concurrency\": 1, \"context_len\": 131072, \"decode_tok_s\": 161.80000000000001136868377216160297393798828125, \"prefill_tok_s\": 5484}, {\"notes\": \"bench card 2026-10-06: ctx_pp 5,236 tok/s @261K; ctx_tg 174.3 tok/s thinking-on @261K; 0.96 s median follow-up first token (one conversation, 26 turns 36K-76K)\", \"ttft_ms\": 960, \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 174.30000000000001136868377216160297393798828125, \"prefill_tok_s\": 5236}, {\"notes\": \"bench card 2026-10-06: aggregate 533.7 tok/s across 8 concurrent (v2.2.0 comparator 531.5); cold prefill median 5,219 tok/s @10K\", \"concurrency\": 8, \"context_len\": 4096, \"decode_tok_s\": 533.700000000000045474735088646411895751953125, \"prefill_tok_s\": 5219}], \"context_len\": 262144}"}],"perf_points":[{"id":2142,"context_len":4096,"prefill_tok_s":5219,"decode_tok_s":533.700000000000045474735088646411895751953125,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"bench card 2026-10-06: aggregate 533.7 tok/s across 8 concurrent (v2.2.0 comparator 531.5); cold prefill median 5,219 tok/s @10K"},{"id":2140,"context_len":131072,"prefill_tok_s":5484,"decode_tok_s":161.80000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"bench card 2026-10-06 @ repo 28cbb72: ctx_pp 5,484 tok/s median @100K depth; ctx_tg 161.8 tok/s @131K thinking-on (ITL 19.85 ms, ~3.13 tokens/event with MTP)"},{"id":2141,"context_len":262144,"prefill_tok_s":5236,"decode_tok_s":174.30000000000001136868377216160297393798828125,"ttft_ms":960,"concurrency":1,"claimed_by":"submitter","notes":"bench card 2026-10-06: ctx_pp 5,236 tok/s @261K; ctx_tg 174.3 tok/s thinking-on @261K; 0.96 s median follow-up first token (one conversation, 26 turns 36K-76K)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin","pin":"17284bf480a624aa6f1338a03c146ce8b2b6acc9 \u2014 4-bit build of Qwen3.8-Flash-Next (Intel AutoRound W4A16 experts + FP8 PLE + MTP draft), FP8-KV-ready config"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/halt95/merlin-engine","pin":"96a0d13b6e98ad4ef86e66a3f6ddb5dd9199a591 \u2014 the repo's vLLM fork: vLLM v0.30.0 + 97 commits (tree 0df6d7ef, verified by build script); source bundle sha256 d904e468\u202637d03d7c"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/halt95/merlin-engine","pin":"28cbb7293f134c244b443fb43bbf6e74ff4d8e24 \u2014 source recipe repo @ HEAD (scripts/serve-v2.5.sh is the verbatim argv source; scripts/build-v2.5.sh, scales/qsa_kv_scales_262k.json, release/, docker-compose.yml all at this commit)"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f.json","alternate":"/r/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:28.596454-06","created_at":"2026-10-06 17:12:38.288062-06","speeds":{"decode_tok_s":168.099999999999994315658113919198513031005859375,"prefill_tok_s":5360,"leaderboard":"/api/v1/leaderboard.json"}}