Qwen3.8-Flash-Next W4A16 2x CMP 170HX TP2+EP MTP4 FP8 PLE 92GB Host

'GPUS=1,2 ./scripts/run.sh' '# docker run (inside run.sh) exec: vllm image entrypoint:' /model --served-model-name flash-next-w4a16 --tensor-parallel-size 2 --enable-expert-parallel --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.92 --kv-cache-dtype auto --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY"}' --speculative-config '{"method":"mtp","num_speculative_tokens":4}' --kv-cache-memory 18897856102 --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --no-enable-flashinfer-autotune --host 0.0.0.0 --port 18024

env: 'GPUS=<two 170HX indices or UUIDs, required> | VLLM_PLE_CPU_OFFLOAD=1 | VLLM_PLE_MODEL_PATH=/ple-model | VLLM_PLE_MANIFEST=/ple-manifest/manifest.json | VLLM_PLE_EMBEDDING_DTYPE=float8_e4m3fn | PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True | VLLM_HTTP_TIMEOUT_KEEP_ALIVE=75 | HF_HOME=/hf-cache | TORCHINDUCTOR_CACHE_DIR=/compile-cache/torchinductor | (docker: --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host --restart unless-stopped)'
notes: "Serve profile = scripts/run.sh @ dc8e0c1e402988c48162425d3cb3c822a48f68bc with scripts/env.sh defaults, transcribed token-for-token (defaults are the Oct 7 heretic-2 retune: batched tokens 8192 + 17.6 GiB KV/card; README Results tables = PRE-update config 1024/24383208960 on the base Vnimanie checkpoint - both recorded). Serves densohax/Qwen3.8-Flash-Next-heretic2-W4A16-FP8PLE @ 705a2ce7f529f61bcd94723bcde66866e0b63286 by default (heretic-2 = trohrbaugh/Qwen3.8-Flash-Next-heretic-2 @ 207cd22c347a, refusals removed; AutoRound W4A16 reworked by prepare-heretic2.sh into the image's AutoGPTQ path: w4 g128 sym, only routed experts quantized (73,728 qweight tensors under .experts.), dynamic -:.*mtp.* keeps MTP head BF16, Marlin experts; base checkpoint's BF16 MTP head grafted as model-mtp-graft.safetensors 4.9 GiB; base tokenizer/chat template byte-identical after rewrite; heretic generation_config stops on 248044 only vs base 248046+248044; repo's own FP8 PLE shards NOT downloaded - locally derived table from the base checkpoint is used). MODEL_DIR=$BASE_MODEL_DIR serves VnimanieAI/Qwen3.8-Flash-Next-W4A16 @ 9236d703b25f25eb5c17e9640204f84fa1ce0c6e (checkpoint every Results-table number was measured on). Expert parallel NOT optional (moe_intermediate_size 640/TP2=320 not divisible by group 128). KV_CACHE_MEMORY= (empty) falls back to gpu-memory-utilization 0.92 sizing (15.5 GiB, 1,016,505 KV tokens, safe side); MAX_NUM_BATCHED_TOKENS=1024 KV_CACHE_MEMORY=24383208960 restores the measured-Results configuration. FP8 KV available via KV_CACHE_DTYPE=fp8_e4m3 + QSA_SCALES=<scales.json> (calibrated example shipped: results/fp8-kv-scales-w4a16-tp2-mtp4.json - MTP drafter's attention layer has no entry in alesha-pro's file so per-composition calibration was required); QSA overlay files mount only in fp8/calibration modes. Measured startup 5 min 50 s (weights 103 s, MTP drafter 9 s, FP8 PLE ~40 s, engine init + CUDA graphs 122 s). Peak disk ~335 GB during preparation (179.8 base + 74 heretic-2 + 51.3 FP8 PLE + 28.8 image; base deletable to ~155 GB). Host RAM: container 61.2 GiB (PLE worker 48.4 GiB) on the 91.9 GiB host; ~64 GiB free required. Cards unmodded: 70 SMs, PCIe Gen2 x16 (lspci LnkSta 5 GT/s x16 - NOT x4; 6.6 GB/s pinned host-device = ~83% of ceiling), no P2P (can_device_access_peer false), HW Power Brake Slowdown Not Active. If a 170HX runs far below these numbers check motherboard PWRBRK# first (cross-ref allover326/deepseek-v4-cmp170hx troubleshooting). Bench method: bench/bench.py streaming, temp 0.7, reasoning_effort medium, decode = (completion-1)/(last-first) excluding TTFT, completion includes reasoning, prefill = prompt/TTFT with random nonce to defeat prefix cache, fresh container per config, median of 3. Chat template defaults reasoning_effort xhigh - xhigh never answered within a 20K budget in 5 of 6 runs (send chat_template_kwargs {\"reasoning_effort\": \"medium\"}; valid xhigh/medium/low, high rejected). Optional LiteLLM gateway config gateway/litellm-config.yaml (motivated the keep-alive fix)."

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-ep-mtp4-fp8-ple-92gb-h.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-ep-mtp4-fp8-ple-92gb-h.json