Qwen3.8-Flash-Next W4A16 3x CMP 170HX PP3 NVMe PLE fp8 KV 1M (clee)

'./run-qwen38fn.sh IMAGE=vllm-sm80:q38fn'

env: "NCCL_P2P_DISABLE=1 (mandatory: no GPU P2P on CMP170HX pairings across root buses) | NCCL_DEBUG=WARN | VLLM_PP_LAYER_PARTITION=17,17,14 (PLE layer idx 1 stays on rank 0) | VLLM_PLE_STAGED=1 (shard-direct preadv PLE; default in run-qwen38fn.sh; without it boot OOMs in allocate_embedding_weight) | VLLM_PLE_MMAP=0 | VLLM_PLE_ZERO=0 (1 = ablation knob that zeroes the PLE contribution) | QWEN4EXP_PLE_MODEL_DIR=/models/qwen38fn-w4a16-autoround (container path; host dir bind-mounted ro at /home/models) | VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 (1M > checkpoint's native 262144) | HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 | PYTORCH_NVML_BASED_CUDA_CHECK=1 VLLM_WORKER_MULTIPROC_METHOD=spawn (verbatim from run-qwen38fn.sh @ 03458ad)"
notes: "All serve args live in configs/intel-1m-fp8-ssmbf16.yml passed as `vllm serve --config /opt/stack.yml` (run-qwen38fn.sh mounts it and execs docker run with --entrypoint vllm); the config resolves to: model /models/qwen38fn-w4a16-autoround, served-model-name qwen3.8, host 0.0.0.0 port 8000 (container) -> host :8080, distributed-executor-backend mp, tensor-parallel-size 1, pipeline-parallel-size 3, disable-custom-all-reduce true (mandatory on this hardware), dtype bfloat16, max-model-len 1048576, block-size 1632 (% 12 == 0 for the QSA ring; do NOT use 8192), max-num-seqs 8, max-num-batched-tokens 8192, gpu-memory-utilization 0.92, kv-cache-dtype fp8_e4m3 (software e4m3 decode on SM80 via the q38fn/fp8-kv-sm80 patch), enable-prefix-caching, enable-chunked-prefill, mamba-cache-mode align, mamba-ssm-cache-dtype bfloat16 (fp32 default eats the fp8 dividend), async-scheduling true, compilation-config cudagraph_mode FULL_AND_PIECEWISE, enable-flashinfer-autotune false, chat-template /opt/chat_template.jinja (froggeric's bugfixed Qwen3.8 templates, mounted by the launcher), trust-request-chat-template true, enable-auto-tool-choice true, tool-call-parser qwen3_xml, reasoning-parser qwen3, enable-prompt-tokens-details true, default-chat-template-kwargs {enable_thinking true, preserve_thinking true, reasoning_effort xhigh}, hf-overrides text_config.layer_types = [linear_attention x3, qwen_sparse_attention] x12 (transformers-5.18 bridge, canonical QSA vocabulary) and rope_parameters {rope_type yarn, factor 4.0, original_max_position_embeddings 262144, rope_theta 10000000, partial_rotary_factor 0.25, mrope_interleaved true, mrope_section [11,11,10]}. Quick start: (1) hf download Intel/Qwen3.8-Flash-Next-W4A16-AutoRound --local-dir /home/models/qwen38fn-w4a16-autoround (>=169 GiB free NVMe); (2) docker pull vllm/vllm-openai:nightly-0cbac6cd1305f710e12193596b27488397bcb205; (3) git clone https://github.com/clee/vllm-sm80 (branches q38fn/*); (4) docker build -t vllm-sm80:q38fn . && ./run-qwen38fn.sh IMAGE=vllm-sm80:q38fn. Healthy boot log check: 'Padding mamba page size by ~2%' — a large or negative percentage means the units-fix patch is not active. Validation: tests/needle.py — exact needle recall at all depths through 1M (d0.5/d0.97/d0.05) plus 17*23=391 deterministic x3, greedy/temperature=0. Troubleshooting: boot OOM in allocate_embedding_weight (cudaHostAlloc) -> VLLM_PLE_STAGED=1 missing; assert attn_page_size >= mamba_page_size -> units-fix patch not mounted; needles pass short-range but fail >=2k -> PLE rows poisoned (VLLM_PLE_ZERO=1 is the ablation knob; QSA attention still carries retrieval); TP2+ NCCL hang -> P2P, see hardware notes; fp8 boot with block 3264 -> units-fix cdiv half missing. Agent-use capability rows: W4A16 experts via GPTQ-Marlin bit-exact; 95.4 GiB BF16 PLE n-gram table streamed LIVE FROM NVMe (VLLM_PLE_STAGED=1, never fully loaded to RAM/VRAM); fp8 e4m3 KV software-decoded on SM80 (q38fn/fp8-kv-sm80); mamba SSM cache bf16 (fp32 default eats the fp8 dividend); ctx 1,048,576 via YaRN x4 (VLLM_ALLOW_LONG_MAX_MODEL_LEN=1; checkpoint native 262144); KV pool 5.97M tokens ~ 5.69 concurrent 1M sessions; froggeric fixed chat templates. Caveats: MTP speculative decoding NOT enabled (draft-layer KV would cost ~16% of the pool; fixes pending in the fork); pin image nightly-0cbac6cd... and fork branch q38fn/base — newer nightlies unvalidated."

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-3x-cmp-170hx-pp3-nvme-ple-fp8-kv-1m-cle.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-3x-cmp-170hx-pp3-nvme-ple-fp8-kv-1m-cle.json