{"slug":"qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6","title":"Qwen3.8-Flash-Next NVFP4 2x CMP 170HX TEP2 GDS SSD PLE MTP6","summary":"Recipe by nguyenthimy2022kg-alt. Community patched vLLM run of the NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on 2x unlocked CMP 170HX (SM80, ~63.4 GiB usable each) with TEP2 tensor+expert parallel and fixed MTP6 speculative decoding with per-request draft isolation. v0.3.0 supports 1-10 active requests (max_num_seqs accepts 1-10, defaults to 1 when omitted; repo example sets 10) with compact/topk vocabulary transport and expanded PCIe communication; history-copy (ngram hybrid) drafting removed. The ~51.2 GB FP8 PLE table stays on NVMe, read GPU-direct via GDS/cuFile with 32 persistent readers and 3 staging slots - no CPU-memory relay; optional host-read PLE backend off by default. FlashInfer PCIe IPC AllReduce, 36-token prefix matching, bayley/cmpunlocker BAR1/P2P driver + host GDS. Measured upstream (2026-10-11, thinking off): 10 requests x 5,000 tokens, 749.67 tok/s aggregate, 76.29/request; historical 2026-09-08 window 151.05-168.29 tok/s decode at 8K-128K inputs xhigh thinking.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":3,"commit_sha":"7d911c7dd2ce6b0a45d3c14791ee74036a43e7ae","content_hash":"247ee93d0bf55ac542edef7091844ac3df34a323","created_at":"2026-10-29 21:32:27.639988-06","notes":"Recipe by nguyenthimy2022kg-alt (community project, Apache-2.0); submitter/listed_by = localai-recipes. Source pinned at 4d229f1438486fa759727f128fc28f3609423dbb (tag v0.3.0, 'feat: release v0.3.0 with ten-request MTP6 and compact transport', 2026-10-14; the v0.2.x-line optional PLE host-read backend f3bfb5d/93f34c8 is retained - default remains SSD-to-GPU GDS, disabled by default). Deploy docs are Chinese; launch argv transcribed verbatim from config/vllm-args.json + scripts/serve.py at 4d229f1438 (single vllm-args delta vs the prior pin: speculative-config num_speculative_tokens 32 -> 6), launch values max_num_seqs=10 / gpu_memory_utilization=0.93 per config/example.json + docs/DEPLOYMENT.en.md; flags verified against serve.py gates (mode tep2/tp2, exactly 2 distinct GPU ids, max_num_seqs 1-10 int, gpu_memory_utilization 0<v<=1, draft_int8 hard-required true, use_local_argmax_reduction tied to it, num_speculative_tokens forced to 6 - 'MTP6 uses six candidates for every request; history copying was removed'; tp2 mode strips --enable-expert-parallel). max_num_seqs accepts 1-10 and defaults to 1 when omitted; the repository example sets it to 10. Requests share the VRAM cache: the 262,144 context limit does not allocate that capacity to all ten requests simultaneously; keep synchronous scheduling. Capability at this pin: ten-request support, fixed MTP6, compact/topk vocabulary transport (NEW src/q38_topk_transport.py, sha256 efa75f6d88bf04c4233ae5dfa9a48547413790b0a950a317de0f0511a4d32796 per patches/runtime-sha256.json, source origin topk-vocab10-validated-20261012); the ngram_hybrid path was removed (src/hybrid-defaults.json + src/ngram_hybrid_runtime.py DELETED). patches/runtime-sha256.json / patches/upstream-sha256.json verified at this pin: 595 runtime files, matching release-validation runtime_files_verified_prepublication=595; model checkpoint revision pin unchanged fc694b54fb0174e0913e6adf86691ef85a4ead47 (per release-validation model.revision) and base image unchanged (Dockerfile diff-empty). Performance: historical 2026-09-08 pre-v0.2.0 RadixArk campaign (README/\u6027\u80fd\u8bb0\u5f55.md) kept, clearly dated; v0.3.0 ten-request numbers authored upstream per docs/release-validation-0.3.0.json + docs/MULTI_REQUEST.en.md (2026-10-11). context_len 2048 in the points array is a short-prompt band marker, not a measured context. The pre-v0.3.0 'window capacity 32 / 16-32-token history fragments (not MTP32)' wording is obsolete at this pin: the history-copy draft path and its num_speculative_tokens=32 window were deleted with ngram_hybrid. kanban t_69ea26f0 (owner go 2026-10-28, preprod only)."},"engine_source":{"type":"fork","git_url":"https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX","base_engine":"vllm","pip_package":null,"base_version":"0.1.dev20073+g8e685d198","image_digest":"docker.io/vllm/vllm-openai@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8","install_note":"NOT a plain pip install: docker build -t qwen-flash-sm80:0.3.0 . (base image vllm/vllm-openai:qwen38-flash-next digest-pinned, cuFile 1.15.1.6-1 + CUDA 13.0 driver-dev added; src/ overlay applied at build; PCIe IPC module compiled, no GPU needed). Host prerequisites the container does NOT configure: bayley/cmpunlocker (5a7bb4b) BAR1/P2P driver + NVMe GDS/P2PDMA validation. Model data: hf download nvidia/Qwen3.8-Flash-Next-NVFP4 @ fc694b54fb0174e0913e6adf86691ef85a4ead47, then check-model.py, prepare-nvidia-view.py, prepare-ple.py mapping/convert-to-gds (--block-rows 256)/enroll. vLLM base version 0.1.dev20073+g8e685d198, PyTorch 2.13 + CUDA 13.0, Triton 3.7.1, Python 3.12. Container RAM limit 21 GiB (+5 swap); needs ~220 GB disk (132.68 GB checkpoint + 51.2 GB PLE + 2.52 GB hardlink view; +80 GB if cross-filesystem).","pinned_commit":"4d229f1438486fa759727f128fc28f3609423dbb"},"credit":[{"name":"nguyenthimy2022kg-alt","url":"https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX"}],"author":{"name":"localai-recipes","isSite":true},"uploaded_by":"localai-recipes","config":{"launch":["docker run -d --pull=never --name <name> --label qwen-flash-sm80.managed=1","--memory 21g --memory-swap 26g --gpus all --ipc=host -p 127.0.0.1:18420:8000","-v <models_root>:/models:ro -v <ple_artifact>:/ple:ro","-v config/cufile.json:/config/cufile.json:ro -v config/local-ple-identity.json:/config/ple-identity.json:ro","-v /run/udev:/run/udev:ro","-e HF_HUB_OFFLINE=1 -e VLLM_PLE_GDS=1 -e VLLM_PLE_GDS_RUNNER=1","-e VLLM_PLE_GDS_WORKERS=32 -e VLLM_PLE_GDS_STAGING_SLOTS=3","-e CUFILE_USE_PCIP2PDMA=1 -e Q38_DRAFT_INT8=1","-e CUDA_VISIBLE_DEVICES=<uuid0>,<uuid1> -e VLLM_PLE_GDS_GPU_UUID=<uuid0> -e VLLM_PLE_GDS_GPU_BDF=<bdf0>","qwen-flash-sm80:0.3.0","vllm serve /models/NVIDIA-Qwen3.8-Flash-Next-GDS","--served-model-name qwen3.8-flash-next --host 0.0.0.0 --port 8000","--load-format safetensors --safetensors-load-strategy lazy","--distributed-executor-backend mp --tensor-parallel-size 2 --pipeline-parallel-size 1 --enable-expert-parallel","--quantization modelopt --linear-backend marlin --moe-backend marlin --dtype bfloat16","--max-model-len 262144 --block-size 3456 --max-num-seqs 10 --gpu-memory-utilization 0.93","--enable-prefix-caching --mamba-cache-mode align --enable-chunked-prefill --max-num-batched-tokens 8192","--no-async-scheduling --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder","--default-chat-template-kwargs {\"enable_thinking\": true, \"preserve_thinking\": true, \"reasoning_effort\": \"xhigh\"}","--override-generation-config {\"temperature\": 1.0, \"top_p\": 0.95, \"top_k\": 20, \"min_p\": 0.0, \"presence_penalty\": 0.0, \"frequency_penalty\": 0.0, \"repetition_penalty\": 1.0, \"max_new_tokens\": 262144}","-cc.cudagraph_mode=FULL_AND_PIECEWISE","-cc.splitting_ops=[\"vllm::unified_attention_with_output\",\"vllm::unified_mla_attention_with_output\",\"vllm::mamba_mixer2\",\"vllm::mamba_mixer\",\"vllm::short_conv\",\"vllm::qwen3_8_flash_next_ple_short_conv\",\"vllm::qwen3_8_flash_next_qsa_with_output\",\"vllm::linear_attention\",\"vllm::qwen_gdn_attention_core\",\"vllm::qwen_gdn_attention_core_fused_norm_packed\",\"vllm::sparse_attn_indexer\",\"vllm::ple_mmap_lookup\",\"vllm::ple_gds_lookup\"]","-cc.inductor_compile_config={\"combo_kernels\":false,\"benchmark_combo_kernel\":false}","--no-enable-flashinfer-autotune","--speculative-config {\"method\": \"mtp\", \"num_speculative_tokens\": 6, \"use_local_argmax_reduction\": true, \"draft_sample_method\": \"greedy\", \"rejection_sample_method\": \"standard\"}","--limit-mm-per-prompt {\"image\":128,\"video\":0} --prefix-match-unit 36"],"layouts":["tp"],"max_context_len":262144,"storage_requirements":{"note":"~220 GB NVMe on the GDS-validated filesystem (132.68 GB checkpoint + 51.2 GB FP8 PLE + 2.52 GB hardlinked view; +80 GB if cross-fs); ~32 GiB host RAM + 8 GiB swap; container capped 21 GiB/26 GiB.","ram_gb":32,"nvme_gb":220}},"models":[{"slug":"qwen3-8-flash-next-nvfp4","name":"Qwen3.8-Flash-Next-NVFP4","hf_repo":"Qwen/Qwen3.8-Flash-Next","quant":"nvfp4"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["cufile","expert-parallel","gds","long-context","marlin","modelopt","mtp6","mtp-speculative","nvfp4","nvme-offload","pcie-ipc","ple","prefix-caching","speculative","ssd-ple","tep2"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"metrics\": {\"notes\": \"Historical 2026-09-08 window on the pre-v0.2.0 RadixArk checkpoint (README table + docs/context-benchmark-20260908.json), xhigh thinking, temperature 1.0/top_p 0.95/top_k 20 streamed decode incl. reasoning tokens; 1 run per length, all natural finish. v0.2.0 (NVIDIA NVFP4 checkpoint) ships a short-prompt 170.826 tok/s natural-completion run (docs/benchmark-summary.json) and a 28-request + 20-turn functional regression; no new long-context table for the NVIDIA checkpoint. Client-side approximations, not isolated GPU prefill. v0.3.0 release run, authored upstream (docs/release-validation-0.3.0.json + docs/MULTI_REQUEST.en.md; benchmark 2026-10-11 America/Los_Angeles, published 2026-10-14): 2x CMP 170HX SM80, TEP2, fixed MTP6, concurrency 10, thinking disabled, temperature=1/top_p=0.95/top_k=20, prompt \u201c\u5199\u4e2a\u7f51\u7ad9\u7f51\u9875\u201d, max_tokens 5000/request, ignore_eos=true; per-request mean 76.29 tok/s (75.19-78.30), aggregate 749.67 tok/s, 50,000 completion tokens, generation cycle 72.34 ms; one screening run per setting, PCIe results from the load immediately following the vocabulary comparison; runtime_image_id sha256:f2ab9f592434fa27b04f78cdd8a7daaa42f873cf993a1be507425f3cbaae5f87. Acceptance: 50 complete text/JSON requests, 2 intentional cancellations, 13 exact checks passed; one extra response ended finish_reason=tool_calls (record kept, not counted as a plain-text speed sample). Ten full-context requests, concurrent multimodal requests and long-duration stress were not rerun on this release.\"}}"}],"perf_points":[{"id":2128,"context_len":2048,"prefill_tok_s":null,"decode_tok_s":170.830000000000012505552149377763271331787109375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"v0.2.0 NVIDIA NVFP4 checkpoint short-prompt natural-completion run (docs/benchmark-summary.json, 2026-09-08); acceptance 0.576, 29010 completion tokens. Short-prompt band marker context_len=2048. ~199 tok/s with thinking off/low effort (README); 259 tok/s sequential-numbers screenshot run."},{"id":2129,"context_len":2048,"prefill_tok_s":null,"decode_tok_s":76.2900000000000062527760746888816356658935546875,"ttft_ms":null,"concurrency":10,"claimed_by":"submitter","notes":"v0.3.0 release run (docs/release-validation-0.3.0.json, benchmark 2026-10-11): 10 concurrent requests x 5,000 tokens (ignore_eos), prompt \u201c\u5199\u4e2a\u7f51\u7ad9\u7f51\u9875\u201d, thinking off, temp=1/top_p=0.95/top_k=20, TEP2+MTP6; per-request mean 76.29 tok/s (75.19-78.30), aggregate 749.67 tok/s, 50,000 tokens, cycle 72.34 ms; runtime_image_id sha256:f2ab9f592434fa27b04f78cdd8a7daaa42f873cf993a1be507425f3cbaae5f87. context_len 2048 = short-prompt band marker."},{"id":2123,"context_len":8192,"prefill_tok_s":1986.3800000000001091393642127513885498046875,"decode_tok_s":168.289999999999992041921359486877918243408203125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"2026-09-08 RadixArk window; input 8193 tok; thinking xhigh; finish stop"},{"id":2124,"context_len":16384,"prefill_tok_s":2083.09999999999990905052982270717620849609375,"decode_tok_s":152.990000000000009094947017729282379150390625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"input 16385 tok"},{"id":2125,"context_len":32768,"prefill_tok_s":2068.84999999999990905052982270717620849609375,"decode_tok_s":156.43999999999999772626324556767940521240234375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"input 32769 tok"},{"id":2126,"context_len":65536,"prefill_tok_s":2036.2100000000000363797880709171295166015625,"decode_tok_s":159.8799999999999954525264911353588104248046875,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"input 65537 tok"},{"id":2127,"context_len":131072,"prefill_tok_s":2059.92999999999983629095368087291717529296875,"decode_tok_s":151.05000000000001136868377216160297393798828125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"input 131073 tok"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4/tree/fc694b54fb0174e0913e6adf86691ef85a4ead47","pin":"fc694b54fb0174e0913e6adf86691ef85a4ead47"},{"kind":"engine_image","pointer_type":"docker_digest","pointer":"https://hub.docker.com/r/vllm/vllm-openai","pin":null},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX","pin":"4d229f1438486fa759727f128fc28f3609423dbb"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6.json","alternate":"/r/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6","describedby":"/openapi.json"},"updated_at":"2026-10-29 21:32:27.639988-06","created_at":"2026-10-20 21:49:18.209897-06","speeds":{"decode_tok_s":158.19999999999998863131622783839702606201171875,"prefill_tok_s":2059.90000000000009094947017729282379150390625,"leaderboard":"/api/v1/leaderboard.json"}}