Qwen3.8-Flash-Next W4A16-Merlin on 4x RTX 3090 — TP2xPP2+EP, MTP3, FP8 KV, 925K pool (halt95)

./venv-v2.5/bin/vllm serve '~/models/Flash-Next-Merlin' --served-model-name flash-next-v2 flash-next flash-mtp flash-next-mtp --host 127.0.0.1 --port 8000 --tensor-parallel-size 2 --pipeline-parallel-size 2 --enable-expert-parallel --speculative-config '{"method":"mtp","num_speculative_tokens":3,"draft_sample_method":"probabilistic"}' --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 1024 --gpu-memory-utilization 0.96 --kv-cache-memory 4100000000 --kv-cache-dtype fp8_e4m3 --limit-mm-per-prompt '{"image":42,"video":0}' --mm-processor-cache-gb 0 --mm-processor-kwargs '{"size":{"longest_edge":4194304,"shortest_edge":65536}}' --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1,2,3,4,6,8,12,16,24,32]}' --enable-prefix-caching --enable-prompt-tokens-details --no-enable-flashinfer-autotune --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' --default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "low"}' --shutdown-timeout 60 --block-size 4096 --use-replayssm --prefix-match-unit 64

env: 'TREE=./vllm-v2.5 VENV=./venv-v2.5 | CUDA_VISIBLE_DEVICES=0,1,2,3 | CUDA_DEVICE_ORDER=PCI_BUS_ID | VLLM_SKIP_P2P_CHECK=1 NCCL_P2P_LEVEL=SYS NCCL_PROTO=LL | VLLM_PP_LAYER_PARTITION=25,23 | VLLM_E2_PLE_PULL_TRANSPORT=1 VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 | VLLM_PLE_OFFLOAD_HOME_DEVICE=0 | VLLM_HOST_EMBED_TABLE=1 | VLLM_USE_BREAKABLE_CUDAGRAPH=0 | VLLM_QSA_KV_SCALES=<repo>/scales/qsa_kv_scales_262k.json (required — without it FP8 KV runs at scale 1.0) | VLLM_CACHE_ROOT=<repo>/.vllm-cache-v2.5 | HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 | MERLIN_FULL_K=1 MERLIN_QSA_SORT=1 MERLIN_TIE_DET=1 MERLIN_TIE_RECENT=1 MERLIN_TOPK_SORTED_EMIT=1 | VLLM_E2_GUARD_MODE=warn | VLLM_E2_SIDE_CACHE_LAYOUT=1 VLLM_USE_V2_MODEL_RUNNER=1 | VLLM_E2_MAMBA_RETAIN_CHECKPOINT=1 | PYTHONPATH=<vllm tree> VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 (all verbatim from scripts/serve-v2.5.sh @ 28cbb72, selector-off legacy path). Opt-in: AUTO_TOPO=1 topology selector; COUNTERS=1 logging-only counters; HOST=0.0.0.0 then set VLLM_API_KEY.'
notes: "Run path: git clone --branch v2.5.1 (repo @ 28cbb72), fetch the v2.5.1 source bundle + the unchanged-since-v2.2.0 compiled-ops tarball from the repo's releases, hf download halt95/Qwen3.8-Flash-Next-W4A16-Merlin --local-dir ~/models/Flash-Next-Merlin (115 GiB), then TREE=./vllm-v2.5 VENV=./venv-v2.5 CACHE_ROOT=./.vllm-cache-v2.5 scripts/serve-v2.5.sh ~/models/Flash-Next-Merlin — the script execs vllm serve with the argv above. Startup ~5-6 min first boot (kernel JIT), 2-3 min after; ready when the log shows 'Application startup complete' and 'GPU KV cache size: 924,993 tokens'. Docker compose variant exists (MODEL_DIR=... docker compose up -d) but the image is not rebuilt/GPU-served on v2.5.1. Requirements: 4x RTX 3090 (sm_86) headless, P2P recommended (runs without at slower prefill), no NVLink; 96 GB host RAM (n-gram table ~48 GiB + ~4.2 GiB pinned embeddings; measured floor ~69 GiB), /dev/shm >= 1 GB, ~250 GB disk. The checkpoint config needs ple_embedding_dtype: float8_e4m3fn inside text_config (present in HF since 2026-09-18). Container needs shm_size 8g + memlock -1 + stop_grace_period 70s. Proxmox LXC recipes in lxc/."

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f.json