localairecipes launch configs for local inference

Qwen3.8-Flash-Next NVFP4 2x CMP 170HX TEP2 GDS SSD PLE MTP6

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by nguyenthimy2022kg-alt↗ · listed by localai-recipes

revision r3 · commit 7d911c7dd2 · updated 21 h ago · license CC-BY-4.0

chat code mtp-speculative long-context speculative prefix-caching nvme-offload nvfp4 modelopt cufile expert-parallel gds marlin mtp6 pcie-ipc ple ssd-ple tep2

Recipe by nguyenthimy2022kg-alt. Community patched vLLM run of the NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on 2x unlocked CMP 170HX (SM80, ~63.4 GiB usable each) with TEP2 tensor+expert parallel and fixed MTP6 speculative decoding with per-request draft isolation. v0.3.0 supports 1-10 active requests (max_num_seqs accepts 1-10, defaults to 1 when omitted; repo example sets 10) with compact/topk vocabulary transport and expanded PCIe communication; history-copy (ngram hybrid) drafting removed. The ~51.2 GB FP8 PLE table stays on NVMe, read GPU-direct via GDS/cuFile with 32 persistent readers and 3 staging slots - no CPU-memory relay; optional host-read PLE backend off by default. FlashInfer PCIe IPC AllReduce, 36-token prefix matching, bayley/cmpunlocker BAR1/P2P driver + host GDS. Measured upstream (2026-10-11, thinking off): 10 requests x 5,000 tokens, 749.67 tok/s aggregate, 76.29/request; historical 2026-09-08 window 151.05-168.29 tok/s decode at 8K-128K inputs xhigh thinking.

Launch command

'docker run -d --pull=never --name <name> --label qwen-flash-sm80.managed=1' '--memory 21g --memory-swap 26g --gpus all --ipc=host -p 127.0.0.1:18420:8000' '-v <models_root>:/models:ro -v <ple_artifact>:/ple:ro' '-v config/cufile.json:/config/cufile.json:ro -v config/local-ple-identity.json:/config/ple-identity.json:ro' '-v /run/udev:/run/udev:ro' '-e HF_HUB_OFFLINE=1 -e VLLM_PLE_GDS=1 -e VLLM_PLE_GDS_RUNNER=1' '-e VLLM_PLE_GDS_WORKERS=32 -e VLLM_PLE_GDS_STAGING_SLOTS=3' '-e CUFILE_USE_PCIP2PDMA=1 -e Q38_DRAFT_INT8=1' '-e CUDA_VISIBLE_DEVICES=<uuid0>,<uuid1> -e VLLM_PLE_GDS_GPU_UUID=<uuid0> -e VLLM_PLE_GDS_GPU_BDF=<bdf0>' qwen-flash-sm80:0.3.0 'vllm serve /models/NVIDIA-Qwen3.8-Flash-Next-GDS' '--served-model-name qwen3.8-flash-next --host 0.0.0.0 --port 8000' '--load-format safetensors --safetensors-load-strategy lazy' '--distributed-executor-backend mp --tensor-parallel-size 2 --pipeline-parallel-size 1 --enable-expert-parallel' '--quantization modelopt --linear-backend marlin --moe-backend marlin --dtype bfloat16' '--max-model-len 262144 --block-size 3456 --max-num-seqs 10 --gpu-memory-utilization 0.93' '--enable-prefix-caching --mamba-cache-mode align --enable-chunked-prefill --max-num-batched-tokens 8192' '--no-async-scheduling --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder' '--default-chat-template-kwargs {"enable_thinking": true, "preserve_thinking": true, "reasoning_effort": "xhigh"}' '--override-generation-config {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "frequency_penalty": 0.0, "repetition_penalty": 1.0, "max_new_tokens": 262144}' -cc.cudagraph_mode=FULL_AND_PIECEWISE '-cc.splitting_ops=["vllm::unified_attention_with_output","vllm::unified_mla_attention_with_output","vllm::mamba_mixer2","vllm::mamba_mixer","vllm::short_conv","vllm::qwen3_8_flash_next_ple_short_conv","vllm::qwen3_8_flash_next_qsa_with_output","vllm::linear_attention","vllm::qwen_gdn_attention_core","vllm::qwen_gdn_attention_core_fused_norm_packed","vllm::sparse_attn_indexer","vllm::ple_mmap_lookup","vllm::ple_gds_lookup"]' '-cc.inductor_compile_config={"combo_kernels":false,"benchmark_combo_kernel":false}' --no-enable-flashinfer-autotune '--speculative-config {"method": "mtp", "num_speculative_tokens": 6, "use_local_argmax_reduction": true, "draft_sample_method": "greedy", "rejection_sample_method": "standard"}' '--limit-mm-per-prompt {"image":128,"video":0} --prefix-match-unit 36'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6.json

Engine

Install note: NOT a plain pip install: docker build -t qwen-flash-sm80:0.3.0 . (base image vllm/vllm-openai:qwen38-flash-next digest-pinned, cuFile 1.15.1.6-1 + CUDA 13.0 driver-dev added; src/ overlay applied at build; PCIe IPC module compiled, no GPU needed). Host prerequisites the container does NOT configure: bayley/cmpunlocker (5a7bb4b) BAR1/P2P driver + NVMe GDS/P2PDMA validation. Model data: hf download nvidia/Qwen3.8-Flash-Next-NVFP4 @ fc694b54fb0174e0913e6adf86691ef85a4ead47, then check-model.py, prepare-nvidia-view.py, prepare-ple.py mapping/convert-to-gds (--block-rows 256)/enroll. vLLM base version 0.1.dev20073+g8e685d198, PyTorch 2.13 + CUDA 13.0, Triton 3.7.1, Python 3.12. Container RAM limit 21 GiB (+5 swap); needs ~220 GB disk (132.68 GB checkpoint + 51.2 GB PLE + 2.52 GB hardlink view; +80 GB if cross-filesystem).

Model

Hardware

Context

  • max context: 256K tokens

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
2,0481—170.8 tok/s—reported
2,04810—76.3 tok/s—reported
8,19211986.4 tok/s168.3 tok/s—reported
16,38412083.1 tok/s153 tok/s—reported
32,76812068.9 tok/s156.4 tok/s—reported
65,53612036.2 tok/s159.9 tok/s—reported
131,07212059.9 tok/s151.1 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

metrics{"notes":"Historical 2026-09-08 window on the pre-v0.2.0 RadixArk checkpoint (README table + docs\/context-benchmark-20260908.json), xhigh thinking, temperature 1.0\/top_p 0.95\/top_k 20 streamed decode incl. reasoning tokens; 1 run per length, all natural finish. v0.2.0 (NVIDIA NVFP4 checkpoint) ships a short-prompt 170.826 tok\/s natural-completion run (docs\/benchmark-summary.json) and a 28-request + 20-turn functional regression; no new long-context table for the NVIDIA checkpoint. Client-side approximations, not isolated GPU prefill. v0.3.0 release run, authored upstream (docs\/release-validation-0.3.0.json + docs\/MULTI_REQUEST.en.md; benchmark 2026-10-11 America\/Los_Angeles, published 2026-10-14): 2x CMP 170HX SM80, TEP2, fixed MTP6, concurrency 10, thinking disabled, temperature=1\/top_p=0.95\/top_k=20, prompt \u201c\u5199\u4e2a\u7f51\u7ad9\u7f51\u9875\u201d, max_tokens 5000\/request, ignore_eos=true; per-request mean 76.29 tok\/s (75.19-78.30), aggregate 749.67 tok\/s, 50,000 completion tokens, generation cycle 72.34 ms; one screening run per setting, PCIe results from the load immediately following the vocabulary comparison; runtime_image_id sha256:f2ab9f592434fa27b04f78cdd8a7daaa42f873cf993a1be507425f3cbaae5f87. Acceptance: 50 complete text\/JSON requests, 2 intentional cancellations, 13 exact checks passed; one extra response ended finish_reason=tool_calls (record kept, not counted as a plain-text speed sample). Ten full-context requests, concurrent multimodal requests and long-duration stress were not rerun on this release."}

Downloads

Revision notes

Recipe by nguyenthimy2022kg-alt (community project, Apache-2.0); submitter/listed_by = localai-recipes. Source pinned at 4d229f1438486fa759727f128fc28f3609423dbb (tag v0.3.0, 'feat: release v0.3.0 with ten-request MTP6 and compact transport', 2026-10-14; the v0.2.x-line optional PLE host-read backend f3bfb5d/93f34c8 is retained - default remains SSD-to-GPU GDS, disabled by default). Deploy docs are Chinese; launch argv transcribed verbatim from config/vllm-args.json + scripts/serve.py at 4d229f1438 (single vllm-args delta vs the prior pin: speculative-config num_speculative_tokens 32 -> 6), launch values max_num_seqs=10 / gpu_memory_utilization=0.93 per config/example.json + docs/DEPLOYMENT.en.md; flags verified against serve.py gates (mode tep2/tp2, exactly 2 distinct GPU ids, max_num_seqs 1-10 int, gpu_memory_utilization 0<v<=1, draft_int8 hard-required true, use_local_argmax_reduction tied to it, num_speculative_tokens forced to 6 - 'MTP6 uses six candidates for every request; history copying was removed'; tp2 mode strips --enable-expert-parallel). max_num_seqs accepts 1-10 and defaults to 1 when omitted; the repository example sets it to 10. Requests share the VRAM cache: the 262,144 context limit does not allocate that capacity to all ten requests simultaneously; keep synchronous scheduling. Capability at this pin: ten-request support, fixed MTP6, compact/topk vocabulary transport (NEW src/q38_topk_transport.py, sha256 efa75f6d88bf04c4233ae5dfa9a48547413790b0a950a317de0f0511a4d32796 per patches/runtime-sha256.json, source origin topk-vocab10-validated-20261012); the ngram_hybrid path was removed (src/hybrid-defaults.json + src/ngram_hybrid_runtime.py DELETED). patches/runtime-sha256.json / patches/upstream-sha256.json verified at this pin: 595 runtime files, matching release-validation runtime_files_verified_prepublication=595; model checkpoint revision pin unchanged fc694b54fb0174e0913e6adf86691ef85a4ead47 (per release-validation model.revision) and base image unchanged (Dockerfile diff-empty). Performance: historical 2026-09-08 pre-v0.2.0 RadixArk campaign (README/性能记录.md) kept, clearly dated; v0.3.0 ten-request numbers authored upstream per docs/release-validation-0.3.0.json + docs/MULTI_REQUEST.en.md (2026-10-11). context_len 2048 in the points array is a short-prompt band marker, not a measured context. The pre-v0.3.0 'window capacity 32 / 16-32-token history fragments (not MTP32)' wording is obsolete at this pin: the history-copy draft path and its num_speculative_tokens=32 window were deleted with ngram_hybrid. kanban t_69ea26f0 (owner go 2026-10-28, preprod only).

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)