{"slug":"qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3","title":"Qwen3.8-Flash-Next W4A16 INT4PLE on 2x modded CMP 170HX \u2014 PP2 MTP3 (DUFRelic)","summary":"Production 2x CMP 170HX recipe for Qwen3.8-Flash-Next: zankich INT4PLE repackage of halt95 Merlin W4A16 (94.4 GiB, 51B-param PLE table repacked INT4) on stock vLLM 0.30.0 + a 6-layer hash-guarded overlay stack (PP2/PLE #56444, KV filter #54793, MTP-over-PP #54709, slot statefix #55506, SpeedSuite 5-PR overlay, batched-P2P). PP2/TP1, MTP 3-token draft (measured 52.27% accept). Hardware is the recipe's own mod stack: 74-SM unlock (cmpunlocker), VBIOS 92.00.6D.00.0A, 170tune 1420 MHz SM cap + HBM 1728 MHz undervolt, BAR1 P2P driver patches, 140 W/card. Author-measured 2026-10-03 (median of 3, isolation audit): prefill C1 6213.7 tok/s @16k, decode C1 133.4 tok/s (SD 22.8 reported), decode C4 agg 245.2 tok/s; 55/56 C at 160 W peaks, no Xid/OOM. INT4 PLE table (~26.8 GiB) pinned in HOST RAM; KV BF16 (pinned build's QSA path rejects FP8). Driver mods optional for function but required for the published throughput.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"c26a7fd62ded418a43f5a10abf82eaedb7afc6bc","content_hash":"7c6051a80e2a07b8bca384eb300935641c88349b","created_at":"2026-10-29 21:32:28.370851-06","notes":"Community ingest of DUFRelic/qwen38-flash-next-recipe @ 25f97f3c (lar-scout GitHub sweep Oct 5, queued as t_dd357cb3; repo created 2026-10-05, 5 commits, README-licensed for reuse; patches and models remain with their upstream authors). Serve profile transcribed from README section 4 @ the pinned commit; benchmark numbers are the author's section 5 tables (median-of-3 methodology with isolation audit), logged claimed_by=submitter T0 \u2014 no run file. Hardware is 'modded CMP 170HX' (74-SM unlock + VBIOS + undervolt + BAR1 P2P) \u2014 mod stack documented in runtime notes and tags (modded-hardware, sm-unlock); profile linked is the existing cmp-170hx-x2 (same silicon, 64 GB x2). First 170hx recipe whose hardware mod stack is part of the recipe. KV cache runs BF16 despite the FP8KV suffix (pinned build rejects FP8 for Qwen4Exp QSA). Model license: qwen-community-1.0."},"engine_source":{"type":"fork","git_url":"https://github.com/DUFRelic/qwen38-flash-next-recipe","base_engine":"vllm","pip_package":null,"base_version":"v0.30.0 @ ced6857a + 6-layer overlay (plugin zankich @ 6609bfef)","image_digest":"sha256:c64a305e9142d6efc7edb5119dc1e25cef46aba8252608abbee4ffc66d575e4a","install_note":"Build: start from vllm/vllm-openai:v0.30.0 @ manifest sha256:5f5e5352..., apply the 6 hash-guarded overlay layers in order (#56444 PP2/PLE, #54793 KV filter, #54709 MTP-over-PP, #55506 statefix @ 5f71d6fa, SpeedSuite #58762/#58763/#58449/#58114/#57097 \u2014 #58449 was OPEN at pin, re-check upstream \u2014 and the batched-P2P rewrite of 3 isend/irecv sites in parallel_state.py); every layer fails closed on hash mismatch. INT4-PLE plugin: zankich/vllm branch v0.29.0z @ 6609bfef (wheel overlay w/ source-hash guard). PUBLIC REBUILD (author's duf/ image name is private): github.com/michael-lgtm/qwen38-flash-next-cmp-overlay apply_overlay.py over the stock image.","pinned_commit":"25f97f3c9c288c19c6c959c6d067005c97261846"},"credit":[{"name":"DUFRelic","url":"https://github.com/DUFRelic/qwen38-flash-next-recipe"},{"name":"zankich (INT4-PLE repackage + plugin)","url":"https://github.com/zankich/vllm"},{"name":"halt95 (W4A16-Merlin base checkpoint)","url":"https://github.com/halt95/merlin-engine"},{"name":"michael-lgtm (public overlay rebuild)","url":"https://github.com/michael-lgtm/qwen38-flash-next-cmp-overlay"},{"name":"amoghmunikote (cmpunlocker SM unlock + BAR1 P2P)","url":"https://github.com/amoghmunikote/cmpunlocker"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/models/qwen38-flash-next-w4a16-merlin-int4ple-fp8kv","--pipeline-parallel-size","2","--tensor-parallel-size","1","--distributed-executor-backend","mp","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":3}","--max-model-len","262144","--max-num-seqs","8","--max-num-batched-tokens","2048","--enable-chunked-prefill","--compilation-config","{\"cudagraph_mode\":\"PIECEWISE\",\"cudagraph_capture_sizes\":[1,2,4,8,16,32]}","--gpu-memory-utilization","0.90","--kv-cache-dtype","auto","--reasoning-parser","qwen3","--tool-call-parser","qwen3_xml","--chat-template","/templates/qwen-fixed-chat-template-v22.5.jinja","--host","127.0.0.1","--port","8000"],"runtime":{"env":"NCCL_P2P_DISABLE=0 | NCCL_P2P_LEVEL=SYS | VLLM_COMPUTE_NANS_IN_LOGITS=1 (diagnostic; per README section 4). KV cache stays BF16 (default) \u2014 the pinned build's Qwen4Exp QSA path rejects an FP8 cache, so --kv-cache-dtype auto is load-bearing. Host-RAM guard: the launcher refuses to start when MemAvailable < 34 GiB (26.8 GiB INT4 PLE table pinned via UVA gather).","notes":"Serve profile transcribed from DUFRelic/qwen38-flash-next-recipe README section 4 @ 25f97f3c (profile p-prefill2048-mtp3-speedsuite) \u2014 this repo documents the profile as a table rather than shipping a serve script. Stack: Caddy/TLS \u2192 LiteLLM \u2192 vLLM (loopback-only) on a Lenovo ThinkStation P520, Docker. Parallelism PP2/TP1 with the mp executor; NOTE from the author: --max-num-seqs 8 under PP2 does NOT create 8 independent in-flight forwards per rank \u2014 the V1 runner caps concurrent batches at the pipeline size. Multimodal: up to 4 images + 1 video (2 frames) per prompt. Chat template: froggeric/Qwen-Fixed-Chat-Templates v22.5 @ 855bffc49448e299789730ff92c9b8d834d6cc14 (chat_template.jinja sha256 e57684ba...), mounted READ-ONLY. Hardware mod stack (verified live, rollback in author's private ops runbooks): VBIOS 92.00.6D.00.0A flashed both cards (max customer boost 1695 MHz, 300 W ceiling); 170tune persistence re-applied every boot (GPC V/F undervolt +250/+275, SM ceiling 1420 MHz via nvidia-smi -lgc 210,1420, HBM NDIV 64 = 1728 MHz, relaxed HBM refresh ~8.3 us); SM unlock 70\u219274 via amoghmunikote/cmpunlocker @ 6c442eeb (0016-sm-reconfig.patch, RECONFIG_PLM + GPC override; verified by CUDA device attributes + %smid across all 74 SMs, 3,145,728 error-free computed values per GPU); BAR1 P2P driver patches (cmpunlocker + Bayley 0011/0013/0015 @ 5a7bb4b7) enabling NCCL P2P SYS. Power: 140 W/card persistent with a 200 W observed-draw abort guard; benchmark telemetry showed 160 W peaks / 55-56 C. Driver mods are OPTIONAL for function (runs with 70 SMs + NCCL_P2P_DISABLE=1) but REQUIRED to match the published throughput. Model files ship pre-packed (model-pleint4-*.safetensors + ple-int4-manifest.json) \u2014 no repacking step. Checkpoint config: Qwen4Exp, ple_layer_ids=[2], ple_embedding_dtype=int4, split_ngram_parts=128, ngram_size=3, heads_per_ngram=8, ple_embed_dim=2560."},"max_context_len":262144},"models":[{"slug":"qwen3-8-flash-next-w4a16-merlin-int4ple-fp8kv","name":"Qwen3.8-Flash-Next-W4A16-Merlin-INT4PLE-FP8KV","hf_repo":"zankich/Qwen3.8-Flash-Next-W4A16-Merlin-INT4PLE-FP8KV","quant":"w4a16-int4-ple-int4"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["dufrelic","int4-ple","modded-hardware","mtp-speculative","p2p","pinned-host-memory","pipeline-parallel","sm-unlock","speedsuite"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured 2026-10-03 (README section 5 @ 25f97f3c, harness llama-benchy on loopback, --pp 16384 --tg 2048 --concurrency 1 4 --runs 3 --exact-tg --no-cache --no-warmup; full-shape warmup excluded from scoring; isolation audit on the scored window: exactly 15 successful requests, 30,720 output tokens, zero prefix-cache hits / preemptions / errors; telemetry every 2 s; stats = mean \u00b1 population SD over 3 scored runs). Prefill C1 @16k 6213.66 \u00b1 99.27; decode C1 (2k out) 133.39 \u00b1 22.81 (min-max 108.10-163.37 \u2014 dispersion reported, not best-run); prefill C4 6639.25 \u00b1 34.79; decode C4 aggregate 245.17 \u00b1 1.17. MTP depth 3 active, draft accept 52.27% over the whole benchmark. 140 W limits held, peaks 159.66/162.77 W, 55/56 C, no Xid/OOM/EngineDeadError. claimed_by=submitter T0 \u2014 no run file ingested; import via POST /api/v1/benchmarks to upgrade.\", \"tok_s\": 133.3899999999999863575794734060764312744140625, \"points\": [{\"notes\": \"README section 5 results table @ 25f97f3c: C1 prefill 6213.66 \u00b1 99.27 / decode (2k out) 133.39 \u00b1 22.81 (min-max 108.10\u2013163.37); MTP3 accept 52.27%; 74 SMs, NCCL-P2P, 140 W, 2026-10-03\", \"concurrency\": 1, \"context_len\": 16384, \"decode_tok_s\": 133.3899999999999863575794734060764312744140625, \"prefill_tok_s\": 6213.65999999999985448084771633148193359375}, {\"notes\": \"README section 5: C4 aggregate of 4 requests 245.17 \u00b1 1.17 (min-max 243.53\u2013246.18); prefill C4 6639.25 \u00b1 34.79\", \"concurrency\": 4, \"context_len\": 16384, \"decode_tok_s\": 245.169999999999987494447850622236728668212890625, \"prefill_tok_s\": 6639.25}], \"context_len\": 16384}"}],"perf_points":[{"id":2138,"context_len":16384,"prefill_tok_s":6213.65999999999985448084771633148193359375,"decode_tok_s":133.3899999999999863575794734060764312744140625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"README section 5 results table @ 25f97f3c: C1 prefill 6213.66 \u00b1 99.27 / decode (2k out) 133.39 \u00b1 22.81 (min-max 108.10\u2013163.37); MTP3 accept 52.27%; 74 SMs, NCCL-P2P, 140 W, 2026-10-03"},{"id":2139,"context_len":16384,"prefill_tok_s":6639.25,"decode_tok_s":245.169999999999987494447850622236728668212890625,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"README section 5: C4 aggregate of 4 requests 245.17 \u00b1 1.17 (min-max 243.53\u2013246.18); prefill C4 6639.25 \u00b1 34.79"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/zankich/Qwen3.8-Flash-Next-W4A16-Merlin-INT4PLE-FP8KV","pin":"45d5db34e05a826d0ca150d4526bc29fd2958d63 \u2014 42 files, 101,361,203,754 bytes (\u224894.4 GiB); repackage of halt95/Qwen3.8-Flash-Next-W4A16-Merlin: 51B-param FP8 PLE table repacked to INT4 (group-32 symmetric) + FP8-KV scales recalibrated; ships model-pleint4-..."},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/vllm/vllm-openai","pin":"base vllm/vllm-openai:v0.30.0 @ manifest sha256:5f5e535216848d0c52159c8c13a0af04be5f6fe1a84e79914300610796f76d40 (commit ced6857afa0ea7b2e3f0846a62e1394e90f15607) + 6-layer hash-guarded overlay; author's final image duf/vllm-030-flash-next-int4ple-pp2:0..."},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/DUFRelic/qwen38-flash-next-recipe","pin":"25f97f3c9c288c19c6c959c6d067005c97261846 \u2014 source recipe repo @ HEAD (README is the recipe: model pins, hardware mod stack, 6-layer patch stack with per-patch SHA-256, serve profile section 4, benchmark methodology section 5)"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/michael-lgtm/qwen38-flash-next-cmp-overlay","pin":"public hash-guarded rebuild of the same 6-layer stack: apply_overlay.py + files/ tree (batched-p2p parallel_state.py rewrite included); use this to reproduce the image without the private duf/ name"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3.json","alternate":"/r/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:28.370851-06","created_at":"2026-10-06 19:35:55.844525-06","speeds":{"decode_tok_s":133.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}