diff --git a/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6/recipe.yaml b/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6/recipe.yaml index 31e0f1bc7..247ee93d0 100644 --- a/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6/recipe.yaml +++ b/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6/recipe.yaml @@ -5,10 +5,10 @@ engine: version: 0.1.dev20073+g8e685d198 source_type: fork git_url: 'https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX' - pinned_commit: 31816b79fbbc69c468171385f89f339f0bd29ff3 + pinned_commit: 4d229f1438486fa759727f128fc28f3609423dbb image_digest: 'docker.io/vllm/vllm-openai@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8' - install_note: 'NOT a plain pip install: docker build -t qwen-flash-sm80:0.2.0 . (base image vllm/vllm-openai:qwen38-flash-next digest-pinned, cuFile 1.15.1.6-1 + CUDA 13.0 driver-dev added; src/ overlay applied at build; PCIe IPC module compiled, no GPU needed). Host prerequisites the container does NOT configure: bayley/cmpunlocker (5a7bb4b) BAR1/P2P driver + NVMe GDS/P2PDMA validation. Model data: hf download nvidia/Qwen3.8-Flash-Next-NVFP4 @ fc694b54fb0174e0913e6adf86691ef85a4ead47, then check-model.py, prepare-nvidia-view.py, prepare-ple.py mapping/convert-to-gds (--block-rows 256)/enroll. vLLM base version 0.1.dev20073+g8e685d198, PyTorch 2.13 + CUDA 13.0, Triton 3.7.1, Python 3.12. Container RAM limit 21 GiB (+5 swap); needs ~220 GB disk (132.68 GB checkpoint + 51.2 GB PLE + 2.52 GB hardlink view; +80 GB if cross-filesystem).' -summary: 'Recipe by nguyenthimy2022kg-alt. Community patched vLLM run of the official NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on 2x unlocked CMP 170HX (SM80, ~63.4 GiB usable each) with TEP2 tensor+expert parallel over the two cards and MTP6 speculative decoding plus 16/32-token historical-text draft fragments (window capacity 32, not MTP32). The ~51.2 GB FP8 PLE table stays on NVMe and is read GPU-direct via GDS/cuFile with 32 persistent readers and 3 staging slots - no CPU-memory relay - overlapping reads with compute; an optional host-read PLE backend for direct-read debugging was added after v0.2.0 (off by default). FlashInfer PCIe IPC AllReduce, 36-token prefix matching, pinned model and software versions, bayley/cmpunlocker driver with BAR1/P2P and host GDS pre-configured. Measured decode 151.05-168.29 tok/s across 8K-128K inputs with xhigh thinking (2026-09-08 window, pre-v0.2.0 RadixArk weights); ~170.8 tok/s short-prompt, ~199 tok/s with low thinking.' + install_note: 'NOT a plain pip install: docker build -t qwen-flash-sm80:0.3.0 . (base image vllm/vllm-openai:qwen38-flash-next digest-pinned, cuFile 1.15.1.6-1 + CUDA 13.0 driver-dev added; src/ overlay applied at build; PCIe IPC module compiled, no GPU needed). Host prerequisites the container does NOT configure: bayley/cmpunlocker (5a7bb4b) BAR1/P2P driver + NVMe GDS/P2PDMA validation. Model data: hf download nvidia/Qwen3.8-Flash-Next-NVFP4 @ fc694b54fb0174e0913e6adf86691ef85a4ead47, then check-model.py, prepare-nvidia-view.py, prepare-ple.py mapping/convert-to-gds (--block-rows 256)/enroll. vLLM base version 0.1.dev20073+g8e685d198, PyTorch 2.13 + CUDA 13.0, Triton 3.7.1, Python 3.12. Container RAM limit 21 GiB (+5 swap); needs ~220 GB disk (132.68 GB checkpoint + 51.2 GB PLE + 2.52 GB hardlink view; +80 GB if cross-filesystem).' +summary: 'Recipe by nguyenthimy2022kg-alt. Community patched vLLM run of the NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on 2x unlocked CMP 170HX (SM80, ~63.4 GiB usable each) with TEP2 tensor+expert parallel and fixed MTP6 speculative decoding with per-request draft isolation. v0.3.0 supports 1-10 active requests (max_num_seqs accepts 1-10, defaults to 1 when omitted; repo example sets 10) with compact/topk vocabulary transport and expanded PCIe communication; history-copy (ngram hybrid) drafting removed. The ~51.2 GB FP8 PLE table stays on NVMe, read GPU-direct via GDS/cuFile with 32 persistent readers and 3 staging slots - no CPU-memory relay; optional host-read PLE backend off by default. FlashInfer PCIe IPC AllReduce, 36-token prefix matching, bayley/cmpunlocker BAR1/P2P driver + host GDS. Measured upstream (2026-10-11, thinking off): 10 requests x 5,000 tokens, 749.67 tok/s aggregate, 76.29/request; historical 2026-09-08 window 151.05-168.29 tok/s decode at 8K-128K inputs xhigh thinking.' license: CC-BY-4.0 model: name: Qwen3.8-Flash-Next-NVFP4 @@ -61,13 +61,13 @@ config: - '-e VLLM_PLE_GDS_WORKERS=32 -e VLLM_PLE_GDS_STAGING_SLOTS=3' - '-e CUFILE_USE_PCIP2PDMA=1 -e Q38_DRAFT_INT8=1' - '-e CUDA_VISIBLE_DEVICES=, -e VLLM_PLE_GDS_GPU_UUID= -e VLLM_PLE_GDS_GPU_BDF=' - - 'qwen-flash-sm80:0.2.0' + - 'qwen-flash-sm80:0.3.0' - 'vllm serve /models/NVIDIA-Qwen3.8-Flash-Next-GDS' - '--served-model-name qwen3.8-flash-next --host 0.0.0.0 --port 8000' - '--load-format safetensors --safetensors-load-strategy lazy' - '--distributed-executor-backend mp --tensor-parallel-size 2 --pipeline-parallel-size 1 --enable-expert-parallel' - '--quantization modelopt --linear-backend marlin --moe-backend marlin --dtype bfloat16' - - '--max-model-len 262144 --block-size 3456 --max-num-seqs 1 --gpu-memory-utilization 0.85' + - '--max-model-len 262144 --block-size 3456 --max-num-seqs 10 --gpu-memory-utilization 0.93' - '--enable-prefix-caching --mamba-cache-mode align --enable-chunked-prefill --max-num-batched-tokens 8192' - '--no-async-scheduling --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder' - '--default-chat-template-kwargs {"enable_thinking": true, "preserve_thinking": true, "reasoning_effort": "xhigh"}' @@ -76,12 +76,12 @@ config: - '-cc.splitting_ops=["vllm::unified_attention_with_output","vllm::unified_mla_attention_with_output","vllm::mamba_mixer2","vllm::mamba_mixer","vllm::short_conv","vllm::qwen3_8_flash_next_ple_short_conv","vllm::qwen3_8_flash_next_qsa_with_output","vllm::linear_attention","vllm::qwen_gdn_attention_core","vllm::qwen_gdn_attention_core_fused_norm_packed","vllm::sparse_attn_indexer","vllm::ple_mmap_lookup","vllm::ple_gds_lookup"]' - '-cc.inductor_compile_config={"combo_kernels":false,"benchmark_combo_kernel":false}' - '--no-enable-flashinfer-autotune' - - '--speculative-config {"method": "mtp", "num_speculative_tokens": 32, "use_local_argmax_reduction": true, "draft_sample_method": "greedy", "rejection_sample_method": "standard"}' + - '--speculative-config {"method": "mtp", "num_speculative_tokens": 6, "use_local_argmax_reduction": true, "draft_sample_method": "greedy", "rejection_sample_method": "standard"}' - '--limit-mm-per-prompt {"image":128,"video":0} --prefix-match-unit 36' performance: claimed_by: submitter metrics: - notes: 'Historical 2026-09-08 window on the pre-v0.2.0 RadixArk checkpoint (README table + docs/context-benchmark-20260908.json), xhigh thinking, temperature 1.0/top_p 0.95/top_k 20 streamed decode incl. reasoning tokens; 1 run per length, all natural finish. v0.2.0 (NVIDIA NVFP4 checkpoint) ships a short-prompt 170.826 tok/s natural-completion run (docs/benchmark-summary.json) and a 28-request + 20-turn functional regression; no new long-context table for the NVIDIA checkpoint. Client-side approximations, not isolated GPU prefill.' + notes: 'Historical 2026-09-08 window on the pre-v0.2.0 RadixArk checkpoint (README table + docs/context-benchmark-20260908.json), xhigh thinking, temperature 1.0/top_p 0.95/top_k 20 streamed decode incl. reasoning tokens; 1 run per length, all natural finish. v0.2.0 (NVIDIA NVFP4 checkpoint) ships a short-prompt 170.826 tok/s natural-completion run (docs/benchmark-summary.json) and a 28-request + 20-turn functional regression; no new long-context table for the NVIDIA checkpoint. Client-side approximations, not isolated GPU prefill. v0.3.0 release run, authored upstream (docs/release-validation-0.3.0.json + docs/MULTI_REQUEST.en.md; benchmark 2026-10-11 America/Los_Angeles, published 2026-10-14): 2x CMP 170HX SM80, TEP2, fixed MTP6, concurrency 10, thinking disabled, temperature=1/top_p=0.95/top_k=20, prompt “写个网站网页”, max_tokens 5000/request, ignore_eos=true; per-request mean 76.29 tok/s (75.19-78.30), aggregate 749.67 tok/s, 50,000 completion tokens, generation cycle 72.34 ms; one screening run per setting, PCIe results from the load immediately following the vocabulary comparison; runtime_image_id sha256:f2ab9f592434fa27b04f78cdd8a7daaa42f873cf993a1be507425f3cbaae5f87. Acceptance: 50 complete text/JSON requests, 2 intentional cancellations, 13 exact checks passed; one extra response ended finish_reason=tool_calls (record kept, not counted as a plain-text speed sample). Ten full-context requests, concurrent multimodal requests and long-duration stress were not rerun on this release.' points: - context_len: 8192 @@ -118,6 +118,11 @@ performance: decode_tok_s: 170.83 concurrency: 1 notes: 'v0.2.0 NVIDIA NVFP4 checkpoint short-prompt natural-completion run (docs/benchmark-summary.json, 2026-09-08); acceptance 0.576, 29010 completion tokens. Short-prompt band marker context_len=2048. ~199 tok/s with thinking off/low effort (README); 259 tok/s sequential-numbers screenshot run.' + - + context_len: 2048 + decode_tok_s: 76.29 + concurrency: 10 + notes: 'v0.3.0 release run (docs/release-validation-0.3.0.json, benchmark 2026-10-11): 10 concurrent requests x 5,000 tokens (ignore_eos), prompt “写个网站网页”, thinking off, temp=1/top_p=0.95/top_k=20, TEP2+MTP6; per-request mean 76.29 tok/s (75.19-78.30), aggregate 749.67 tok/s, 50,000 tokens, cycle 72.34 ms; runtime_image_id sha256:f2ab9f592434fa27b04f78cdd8a7daaa42f873cf993a1be507425f3cbaae5f87. context_len 2048 = short-prompt band marker.' artifacts: - kind: model_weights @@ -127,12 +132,13 @@ artifacts: - kind: engine_image pointer_type: docker_digest - pointer: 'https://hub.docker.com/vllm/vllm-openai@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8' + pointer: 'https://hub.docker.com/r/vllm/vllm-openai' + pin: null - kind: reference pointer_type: git_commit pointer: 'https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX' - pin: 31816b79fbbc69c468171385f89f339f0bd29ff3 + pin: 4d229f1438486fa759727f128fc28f3609423dbb provenance: credit: - @@ -140,5 +146,5 @@ provenance: url: 'https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX' source_urls: - 'https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX' - notes: "Recipe by nguyenthimy2022kg-alt (community project, Apache-2.0); submitter/listed_by = localai-recipes. Source pinned at git HEAD 31816b79fbbc69c468171385f89f339f0bd29ff3 (tag v0.2.0 = 11421471c050cccc036b004e91345583e33993b2, pushed 2026-10-10; HEAD adds commits f3bfb5d/93f34c8: OPTIONAL PLE host-read backend for sync_mt direct-read debugging - default remains SSD-to-GPU GDS, disabled by default, requires rebuild). Deploy docs are Chinese; launch argv transcribed from config/vllm-args.json + scripts/serve.py + docs/DEPLOYMENT.en.md at the pin, flags verified against serve.py gates (mode tep2/tp2, exactly 2 distinct GPU ids, draft_int8 hard-required true in v0.2.0, use_local_argmax_reduction tied to it; tp2 mode strips --enable-expert-parallel). Performance numbers are the 2026-09-08 pre-v0.2.0 RadixArk campaign (README/性能记录.md); kept as-of honest per card. context_len 2048 in the points array is a marker for the short-prompt band, not a measured context. 'MTP6' = MTP depth 6 per repo docs; speculative window num_speculative_tokens=32 is the draft capacity for 16/32-token history fragments (repo: 'not MTP32'). v0.2.0 regression: 28-request + 20-turn same-session regression incl. ~260K input" + notes: "Recipe by nguyenthimy2022kg-alt (community project, Apache-2.0); submitter/listed_by = localai-recipes. Source pinned at 4d229f1438486fa759727f128fc28f3609423dbb (tag v0.3.0, 'feat: release v0.3.0 with ten-request MTP6 and compact transport', 2026-10-14; the v0.2.x-line optional PLE host-read backend f3bfb5d/93f34c8 is retained - default remains SSD-to-GPU GDS, disabled by default). Deploy docs are Chinese; launch argv transcribed verbatim from config/vllm-args.json + scripts/serve.py at 4d229f1438 (single vllm-args delta vs the prior pin: speculative-config num_speculative_tokens 32 -> 6), launch values max_num_seqs=10 / gpu_memory_utilization=0.93 per config/example.json + docs/DEPLOYMENT.en.md; flags verified against serve.py gates (mode tep2/tp2, exactly 2 distinct GPU ids, max_num_seqs 1-10 int, gpu_memory_utilization 0