diff --git a/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6/recipe.yaml b/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6/recipe.yaml index 9e9df1140..e7984fad9 100644 --- a/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6/recipe.yaml +++ b/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6/recipe.yaml @@ -8,7 +8,7 @@ engine: pinned_commit: 31816b79fbbc69c468171385f89f339f0bd29ff3 image_digest: 'docker.io/vllm/vllm-openai@sha256:fc120ece0a388cc0aa1caad4a9f1cd92113484ab7ec2fd0efadd62585be05bf8' install_note: 'NOT a plain pip install: docker build -t qwen-flash-sm80:0.2.0 . (base image vllm/vllm-openai:qwen38-flash-next digest-pinned, cuFile 1.15.1.6-1 + CUDA 13.0 driver-dev added; src/ overlay applied at build; PCIe IPC module compiled, no GPU needed). Host prerequisites the container does NOT configure: bayley/cmpunlocker (5a7bb4b) BAR1/P2P driver + NVMe GDS/P2PDMA validation. Model data: hf download nvidia/Qwen3.8-Flash-Next-NVFP4 @ fc694b54fb0174e0913e6adf86691ef85a4ead47, then check-model.py, prepare-nvidia-view.py, prepare-ple.py mapping/convert-to-gds (--block-rows 256)/enroll. vLLM base version 0.1.dev20073+g8e685d198, PyTorch 2.13 + CUDA 13.0, Triton 3.7.1, Python 3.12. Container RAM limit 21 GiB (+5 swap); needs ~220 GB disk (132.68 GB checkpoint + 51.2 GB PLE + 2.52 GB hardlink view; +80 GB if cross-filesystem).' -summary: 'Recipe by nguyenthimy2022kg-alt. Community patched vLLM run of the official NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on 2x unlocked CMP 170HX (SM80, ~63.4 GiB usable each) with TEP2 tensor+expert parallel over the two cards and MTP6 speculative decoding plus 16/32-token historical-text draft fragments (window capacity 32, not MTP32). The ~51.2 GB FP8 PLE table stays on NVMe and is read GPU-direct via GDS/cuFile with 32 persistent readers and 3 staging slots - no CPU-memory relay - overlapping reads with compute; an optional host-read PLE backend for direct-read debugging was added after v0.2.0 (off by default). FlashInfer PCIe IPC AllReduce, 36-token prefix matching, pinned model and software versions, bayley/cmpunlocker driver with BAR1/P2P and host GDS pre-configured. Measured decode 151.05-168.29 tok/s across 8K-128K inputs with xhigh thinking (2026-09-08 window, pre-v0.2.0 RadixArk weights); ~170.8 tok/s short-prompt, ~199 tok/s with low thinking..' +summary: 'Recipe by nguyenthimy2022kg-alt. Community patched vLLM run of the official NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on 2x unlocked CMP 170HX (SM80, ~63.4 GiB usable each) with TEP2 tensor+expert parallel over the two cards and MTP6 speculative decoding plus 16/32-token historical-text draft fragments (window capacity 32, not MTP32). The ~51.2 GB FP8 PLE table stays on NVMe and is read GPU-direct via GDS/cuFile with 32 persistent readers and 3 staging slots - no CPU-memory relay - overlapping reads with compute; an optional host-read PLE backend for direct-read debugging was added after v0.2.0 (off by default). FlashInfer PCIe IPC AllReduce, 36-token prefix matching, pinned model and software versions, bayley/cmpunlocker driver with BAR1/P2P and host GDS pre-configured. Measured decode 151.05-168.29 tok/s across 8K-128K inputs with xhigh thinking (2026-09-08 window, pre-v0.2.0 RadixArk weights); ~170.8 tok/s short-prompt, ~199 tok/s with low thinking.' license: CC-BY-4.0 model: name: Qwen3.8-Flash-Next-NVFP4