{"slug":"qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-ep-mtp4-fp8-ple-92gb-h","title":"Qwen3.8-Flash-Next W4A16 2x CMP 170HX TP2+EP MTP4 FP8 PLE 92GB Host","summary":"vLLM recipe for plain Qwen3.8-Flash-Next W4A16 on two STOCK CMP 170HX (Gen2 x16, no P2P) on a LOW-RAM 92 GiB host. Differentiator: the 95.4 GiB BF16 PLE table is converted LOCALLY to FP8 E4M3 (amax/448 global scale) into alesha-pro's offload-overlay layout - scale bit-identical to RadixArk's published FP8 PLE, no second 180 GB download - halving host residency to 47.7 GiB. Serves refusal-removed heretic-2 (densohax W4A16) via the image's AutoGPTQ path (BF16 MTP head grafted); MODEL_DIR=$BASE_MODEL_DIR serves the Vnimanie base. Pinned vLLM image + alesha-pro overlays read-only. TP2 + expert-parallel REQUIRED (MoE 640/2 fails group 128), MTP k=4, BF16 KV, seqs 8, batched tokens 8192 + 17.6 GiB KV/card (Oct 7 retune), 262K ctx. Measured (base ckpt, median of 3): decode 104.5 ja / 101.1 en / 114.0 list / 173 code @64K, 392 agg @4 conc, prefill 2637-3068, 200K needle 11/12. Negatives: FP8 KV -6..-11% decode, PP unsupported by PLE offload, 4096 batched tokens 500s, xhigh trap (use medium).","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"e55df425b63abacf9956879ff28dc971ac4b43e6","content_hash":"8f83b7a6e39e15a80523703eeb5012f05282838a","created_at":"2026-10-20 15:34:14.170637-06","notes":"Ingested from GitHub repo moriyasujapan/qwen38-flash-next-vllm-2x170hx @ dc8e0c1e402988c48162425d3cb3c822a48f68bc (lar-scout ingest, kanban t_cbb08e7d). Repo created 2026-09-21; Oct 7 commit made heretic-2 the default serve target and retuned defaults to batched tokens 8192 + 17.6 GiB KV - launch transcribed from scripts @ HEAD, NOT from README tables (which document the pre-update config on the base checkpoint; both recorded here). 4th 2x CMP 170HX Qwen3.8-Flash-Next recipe on site: distinct checkpoint row (VnimanieAI plain W4A16 vs mannydelrio Uncensored-AWQ / DUFRelic INT4PLE / bozocentral INT4-Mixed / bluespace3 albucino W4A16-FP8PLE), distinct topology (TP2+expert-parallel vs the PP2 builds), distinct trick (LOCAL FP8 conversion of the PLE table with RadixArk-bit-identical scale instead of off-the-shelf quant checkpoints), distinct constraint story (92 GiB host RAM). Recipe code Apache-2.0; weights under the Qwen Community License (no weights in repo). Dedupe nuance flagged to owner at ingest: 4th/5th 2x170HX flash-next recipe - droppable on owner word if coverage considered saturated."},"engine_source":{"type":"docker_image","git_url":"https://github.com/moriyasujapan/qwen38-flash-next-vllm-2x170hx","base_engine":"vllm","pip_package":null,"base_version":"qwen38-flash-next image + alesha-pro overlays @ 9b36aaec","image_digest":"vllm/vllm-openai:qwen38-flash-next-x86_64-cu130@sha256:0aea30240f3e3d9ffae8526643950e170eb5fa07fc427016a9dd90892afa2aa3","install_note":"setup.sh @ dc8e0c1e: clone alesha-pro/qwen38-flash-next-4x3090 into vendor/ @ 9b36aaec3f604d726c6b751e29f7de8225fc203b (env.sh OVERLAY_REV); docker pull vllm/vllm-openai:qwen38-flash-next-x86_64-cu130 (digest sha256:0aea30240f... live 2026-10-08). run.sh mounts 8 overlays read-only INTO the image's vLLM tree (ple_layer.py, ple_offload/worker.py + ple_external_source.py, gpu_worker.py, multiproc_executor.py, qwen_gdn_linear_attn.py, shared_experts.py; +qsa.py/ops_qsa.py fp8-KV only). Builds: download-model.sh, build-ple-fp8.sh -> quantize_ple_fp8.py (BF16 PLE -> FP8 E4M3 + manifest), download-heretic2.sh, prepare_heretic2.py (GPTQ rewrite + MTP graft). Serve: GPUS=<idx/UUIDs> ./scripts/run.sh. Launch token-for-token from run.sh + env.sh @ dc8e0c1e: len 262144, seqs 8, batched 8192 (Oct 7 retune), util 0.92, KV auto, kv-mem 18897856102, spec mtp k4; docker --ipc=host --cap-add SYS_PTRACE --security-opt seccomp=unconfined. Full env+flag dump in config.runtime.env.","pinned_commit":"dc8e0c1e402988c48162425d3cb3c822a48f68bc"},"credit":[{"name":"moriyasujapan (recipe, local FP8 PLE conversion, measurements)","url":"https://github.com/moriyasujapan/qwen38-flash-next-vllm-2x170hx"},{"name":"alesha-pro/qwen38-flash-next-4x3090 (PLE offload overlays + manifest tooling, pinned not vendored)","url":"https://github.com/alesha-pro/qwen38-flash-next-4x3090"},{"name":"VnimanieAI (base W4A16 quantization)","url":"https://huggingface.co/VnimanieAI/Qwen3.8-Flash-Next-W4A16"},{"name":"trohrbaugh (heretic-2 refusals-removed checkpoint) + densohax (its W4A16 quantization)","url":"https://huggingface.co/densohax/Qwen3.8-Flash-Next-heretic2-W4A16-FP8PLE"},{"name":"loktar00/qwen38-flash-next-vllm-3090-recipe (starting settings + MTP findings)","url":"https://github.com/loktar00/qwen38-flash-next-vllm-3090-recipe"},{"name":"RadixArk/Qwen3.8-Flash-Next-NVFP4 (FP8 PLE layout reproduced bit-identically)","url":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["GPUS=1,2 ./scripts/run.sh","# docker run (inside run.sh) exec: vllm image entrypoint:","/model","--served-model-name","flash-next-w4a16","--tensor-parallel-size","2","--enable-expert-parallel","--max-model-len","262144","--max-num-seqs","8","--max-num-batched-tokens","8192","--gpu-memory-utilization","0.92","--kv-cache-dtype","auto","--compilation-config","{\"mode\":0,\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":4}","--kv-cache-memory","18897856102","--enable-auto-tool-choice","--tool-call-parser","qwen3_xml","--reasoning-parser","qwen3","--no-enable-flashinfer-autotune","--host","0.0.0.0","--port","18024"],"runtime":{"env":"GPUS=<two 170HX indices or UUIDs, required> | VLLM_PLE_CPU_OFFLOAD=1 | VLLM_PLE_MODEL_PATH=/ple-model | VLLM_PLE_MANIFEST=/ple-manifest/manifest.json | VLLM_PLE_EMBEDDING_DTYPE=float8_e4m3fn | PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | VLLM_HTTP_TIMEOUT_KEEP_ALIVE=75 | HF_HOME=/hf-cache | TORCHINDUCTOR_CACHE_DIR=/compile-cache/torchinductor | (docker: --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host --restart unless-stopped)","notes":"Serve profile = scripts/run.sh @ dc8e0c1e402988c48162425d3cb3c822a48f68bc with scripts/env.sh defaults, transcribed token-for-token (defaults are the Oct 7 heretic-2 retune: batched tokens 8192 + 17.6 GiB KV/card; README Results tables = PRE-update config 1024/24383208960 on the base Vnimanie checkpoint - both recorded). Serves densohax/Qwen3.8-Flash-Next-heretic2-W4A16-FP8PLE @ 705a2ce7f529f61bcd94723bcde66866e0b63286 by default (heretic-2 = trohrbaugh/Qwen3.8-Flash-Next-heretic-2 @ 207cd22c347a, refusals removed; AutoRound W4A16 reworked by prepare-heretic2.sh into the image's AutoGPTQ path: w4 g128 sym, only routed experts quantized (73,728 qweight tensors under .experts.), dynamic -:.*mtp.* keeps MTP head BF16, Marlin experts; base checkpoint's BF16 MTP head grafted as model-mtp-graft.safetensors 4.9 GiB; base tokenizer/chat template byte-identical after rewrite; heretic generation_config stops on 248044 only vs base 248046+248044; repo's own FP8 PLE shards NOT downloaded - locally derived table from the base checkpoint is used). MODEL_DIR=$BASE_MODEL_DIR serves VnimanieAI/Qwen3.8-Flash-Next-W4A16 @ 9236d703b25f25eb5c17e9640204f84fa1ce0c6e (checkpoint every Results-table number was measured on). Expert parallel NOT optional (moe_intermediate_size 640/TP2=320 not divisible by group 128). KV_CACHE_MEMORY= (empty) falls back to gpu-memory-utilization 0.92 sizing (15.5 GiB, 1,016,505 KV tokens, safe side); MAX_NUM_BATCHED_TOKENS=1024 KV_CACHE_MEMORY=24383208960 restores the measured-Results configuration. FP8 KV available via KV_CACHE_DTYPE=fp8_e4m3 + QSA_SCALES=<scales.json> (calibrated example shipped: results/fp8-kv-scales-w4a16-tp2-mtp4.json - MTP drafter's attention layer has no entry in alesha-pro's file so per-composition calibration was required); QSA overlay files mount only in fp8/calibration modes. Measured startup 5 min 50 s (weights 103 s, MTP drafter 9 s, FP8 PLE ~40 s, engine init + CUDA graphs 122 s). Peak disk ~335 GB during preparation (179.8 base + 74 heretic-2 + 51.3 FP8 PLE + 28.8 image; base deletable to ~155 GB). Host RAM: container 61.2 GiB (PLE worker 48.4 GiB) on the 91.9 GiB host; ~64 GiB free required. Cards unmodded: 70 SMs, PCIe Gen2 x16 (lspci LnkSta 5 GT/s x16 - NOT x4; 6.6 GB/s pinned host-device = ~83% of ceiling), no P2P (can_device_access_peer false), HW Power Brake Slowdown Not Active. If a 170HX runs far below these numbers check motherboard PWRBRK# first (cross-ref allover326/deepseek-v4-cmp170hx troubleshooting). Bench method: bench/bench.py streaming, temp 0.7, reasoning_effort medium, decode = (completion-1)/(last-first) excluding TTFT, completion includes reasoning, prefill = prompt/TTFT with random nonce to defeat prefix cache, fresh container per config, median of 3. Chat template defaults reasoning_effort xhigh - xhigh never answered within a 20K budget in 5 of 6 runs (send chat_template_kwargs {\"reasoning_effort\": \"medium\"}; valid xhigh/medium/low, high rejected). Optional LiteLLM gateway config gateway/litellm-config.yaml (motivated the keep-alive fix)."},"max_context_len":262144,"storage_requirements":{"ram_gb":92}},"models":[{"slug":"qwen3-8-flash-next-w4a16","name":"Qwen3.8-Flash-Next-W4A16","hf_repo":"VnimanieAI/Qwen3.8-Flash-Next-W4A16","quant":"w4a16"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["bf16-kv","docker-image","fp8-ple-offload","heretic-uncensored","low-ram-host","moriyasujapan","mtp-speculative","ple-conversion","sm80","tp2-expert-parallel"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured on the BASE Vnimanie checkpoint with the then-default config (262144 ctx, MTP k=4, max-num-seqs 8, batched tokens 1024, BF16 KV, kv-cache-memory 24383208960; 64K rows measured at --max-model-len 65536 before the default moved) - README Results tables + results/2026-09-21-sweep.jsonl @ dc8e0c1e402988c48162425d3cb3c822a48f68bc, 2026-09-21. claimed_by=submitter T0 - no run file ingested; import via POST /api/v1/benchmarks to upgrade. Heretic-2 (current default serve target) measured separately (bench-of-us depth ladder, PRE-retune defaults): 32K prefill 3323 / decode 186.2 synthetic (accept 0.94) / 97.1 real (0.39); 128K 2994/184.6/89.1; 258K 2575/178.3/69.7 - synthetic-vs-real 2.6x gap = MTP acceptance alone; NOT comparable with the base bench.py tables.\", \"tok_s\": 173, \"points\": [{\"notes\": \"README decode table (ja prose, MTP k=4, max-num-seqs 8, median of 3): 104.5; en 101.1; ja list 114.0; code 173.0 (64K rows @ --max-model-len 65536)\", \"concurrency\": 1, \"context_len\": 65536, \"decode_tok_s\": 104.5}, {\"notes\": \"README decode table at the 262K default: ja 101.3 / en 97.8 / ja list 107.7 / code 170.0 (same MTP k=4 seqs 8 config)\", \"concurrency\": 1, \"context_len\": 262144, \"decode_tok_s\": 101.2999999999999971578290569595992565155029296875}, {\"notes\": \"README decode table no-MTP baseline: ja 65.8 / en 65.9 / ja list 64.9 / code 66.5 (MTP worth ~1.6x prose ~2.6x code; k2 105.3/102.2/101.2/133.6, k3 105.9/104.6/111.1/156.5, k4-seqs1 code 182.0)\", \"concurrency\": 1, \"context_len\": 65536, \"decode_tok_s\": 66.5}, {\"notes\": \"README concurrency table (aggregate tok/s, 256-token code answers, max-num-seqs 8): c4 333, c8 302, c1 127 (64K rows); at the 262K default c4 392 / c8 387 / c1 126\", \"concurrency\": 4, \"context_len\": 65536, \"decode_tok_s\": 333}, {\"notes\": \"README concurrency table at the 262K default: c4 392 aggregate (max-num-seqs 8, 256-token code answers)\", \"concurrency\": 4, \"context_len\": 262144, \"decode_tok_s\": 392}, {\"notes\": \"README prefill table (prefix cache defeated w/ random nonce): 6954-tok prompt 2637 tok/s / 27853-tok 3068 tok/s @64K\", \"concurrency\": 1, \"context_len\": 65536, \"prefill_tok_s\": 2637}, {\"notes\": \"README prefill table at the 262K default: 6954-tok 3191 tok/s / 27853-tok 3261 tok/s\", \"concurrency\": 1, \"context_len\": 262144, \"prefill_tok_s\": 3191}, {\"notes\": \"README long-context: 200,087-token chat prompt, code needle planted at 24% depth, returned correctly; TTFT 70.5 s (= 2839 tok/s prefill), then 69.0 tok/s decode at that depth; repeated prefixes served from prefix cache\", \"ttft_ms\": 70500, \"concurrency\": 1, \"context_len\": 200087, \"decode_tok_s\": 69, \"prefill_tok_s\": 2839}], \"context_len\": 262144}"}],"perf_points":[{"id":1062,"context_len":65536,"prefill_tok_s":null,"decode_tok_s":104.5,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"README decode table (ja prose, MTP k=4, max-num-seqs 8, median of 3): 104.5; en 101.1; ja list 114.0; code 173.0 (64K rows @ --max-model-len 65536)"},{"id":1064,"context_len":65536,"prefill_tok_s":null,"decode_tok_s":333,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"README concurrency table (aggregate tok/s, 256-token code answers, max-num-seqs 8): c4 333, c8 302, c1 127 (64K rows); at the 262K default c4 392 / c8 387 / c1 126"},{"id":1066,"context_len":200087,"prefill_tok_s":2839,"decode_tok_s":69,"ttft_ms":70500,"concurrency":1,"claimed_by":"submitter","notes":"README long-context: 200,087-token chat prompt, code needle planted at 24% depth, returned correctly; TTFT 70.5 s (= 2839 tok/s prefill), then 69.0 tok/s decode at that depth; repeated prefixes served from prefix cache"},{"id":1063,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":101.2999999999999971578290569595992565155029296875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"README decode table at the 262K default: ja 101.3 / en 97.8 / ja list 107.7 / code 170.0 (same MTP k=4 seqs 8 config)"},{"id":1065,"context_len":262144,"prefill_tok_s":null,"decode_tok_s":392,"ttft_ms":null,"concurrency":4,"claimed_by":"submitter","notes":"README concurrency table at the 262K default: c4 392 aggregate (max-num-seqs 8, 256-token code answers)"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-ep-mtp4-fp8-ple-92gb-h.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-ep-mtp4-fp8-ple-92gb-h.json"},"requirements":{"requires":["Docker (or a compatible OCI container runtime)","NVIDIA Container Toolkit (nvidia-container-toolkit) - exposes host GPUs to the container"],"driver_floor":{"cuda":null,"linux_min":null,"text":"unknown - check the image tag","source":"NVIDIA CUDA Toolkit Release Notes - driver compatibility tables (https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/, fetched 2026-10-03)"},"placeholders":["Hardware-bound values in this recipe (device counts, bind-mount paths, utilization, ports) describe the author's machine, not yours - map them to your own rig before launching. Keep the engine flags, model, quant, and parallelism strategy; adjust paths and ports to the hardware you actually have.","GPU identity: list your GPUs with `nvidia-smi -L` and substitute your own GPU UUIDs/indices wherever the recipe pins the author's.","Model directory: bind-mount paths are author-side - point them at your own mount where the model files live.","Host port: pick a free port on your machine and publish the container there; keep the in-container port unchanged."]},"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/moriyasujapan/qwen38-flash-next-vllm-2x170hx","pin":"dc8e0c1e402988c48162425d3cb3c822a48f68bc - recipe repo @ HEAD 2026-10-07 (scripts/, bench/, gateway/, results/, READMEs; run.sh+env.sh+setup.sh+quantize_ple_fp8.py+prepare_heretic2.*; Apache-2.0)"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/VnimanieAI/Qwen3.8-Flash-Next-W4A16","pin":"revision 9236d703b25f25eb5c17e9640204f84fa1ce0c6e (base W4A16 179.8 GB; source of BF16 PLE table, BF16 MTP head + tokenizer; MODEL_DIR=$BASE_MODEL_DIR serves it directly; every Results-table number measured on it)"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/densohax/Qwen3.8-Flash-Next-heretic2-W4A16-FP8PLE","pin":"revision 705a2ce7f529f61bcd94723bcde66866e0b63286 (heretic-2 AutoRound W4A16, 74 GB, sha256-verified; DEFAULT serve target after prepare-heretic2.sh; of trohrbaugh/Qwen3.8-Flash-Next-heretic-2 @ 207cd22c347a refusals removed)"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/vllm/vllm-openai","pin":"vllm/vllm-openai:qwen38-flash-next-x86_64-cu130 @ sha256:0aea30240f3e3d9ffae8526643950e170eb5fa07fc427016a9dd90892afa2aa3 (manifest digest resolved live 2026-10-08; tag updated 2026-08-26, ~8.6 GB; CUDA 13.0)"},{"kind":"reference","pointer_type":"url","pointer":"https://github.com/alesha-pro/qwen38-flash-next-4x3090","pin":"commit 9b36aaec3f604d726c6b751e29f7de8225fc203b (env.sh OVERLAY_REV; PLE host-offload overlays + manifest tooling + QSA fp8-KV path; pinned not vendored per README credits)"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-ep-mtp4-fp8-ple-92gb-h.json","alternate":"/r/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-ep-mtp4-fp8-ple-92gb-h","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:14.170637-06","created_at":"2026-10-07 19:43:54.57324-06","speeds":{"decode_tok_s":101.2999999999999971578290569595992565155029296875,"prefill_tok_s":2839,"leaderboard":"/api/v1/leaderboard.json"}}