{"slug":"qwen3-8-flash-next-uncensored-awq-g32-on-3x-cmp-170hx-pp3-1m-ctx","title":"Qwen3.8-Flash-Next Uncensored AWQ-g32 on 3x CMP 170HX \u2014 PP3, 1M ctx, NVMe PLE, MTP-4 (givisok)","summary":"MTP-4 production recipe for Qwen3.8-Flash-Next-Uncensored (leoncca AWQ-g32) on 3x CMP 170HX (SM80 cards unlocked to 64 GiB): PP3 partition 17,17,14 (no GPU P2P on this pairing; the last rank carries target lm_head + MTP draft and binds the KV pool by raw bytes), fp8_e4m3 KV software-decoded on SM80, bf16 mamba SSM cache, 1,048,576-token context via YaRN x4 over the 262,144-native checkpoint, 51 GB fp8 PLE table streamed from NVMe (shard-direct preadv), MTP-4 speculative decode with the author's mtp-fp8 draft + W4A16 GPTQ-Marlin draft head. KV pool 5.28M tokens (5.04x) at gpu-mem-util 0.95. 10 required fork patches (QSA top-k tie fix, fp8 PLE staging, int8 PP hop, FN12 bounded prefill, ...); engine = clee vllm-sm80 q38fn lineage on pinned nightly 7d0b4e57, no published image. Author-claimed 2026-11-02: solo code 195 / structured 241 / prose 149 tok/s, decode 134 tok/s @1M (1.63 acc/draft), N=8 code aggregate 686, cold prefill 4,287 tok/s @1M, 18/18 multi-needle PASS (5-95% depth).","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":1,"commit_sha":"37840ba5bb894528407c36c2c21873dcbf65fb79","content_hash":"bdd26bdca3f3b757857bbd9fec477735e3e19ed9","created_at":"2026-11-02 03:44:32.301706-07","notes":"Submitted 2026-11-02 by localai-recipes (poster/listed_by per attribution rule) via lar-scout ingest cards t_a7e81981 + full-body twin t_481f538a (parent digest t_430360a8). Repo @ 91803a4a18a1d96d4d73ed1709b11937e6bd1f63 (HEAD verified via git ls-remote 2026-11-02). Launch argv + env transcribed verbatim from scripts/run-qwen38fn.sh + configs/awq-1m-fp8-ssmbf16-mtp4.yml (launcher default since 2026-11-02) at that sha; all serve args live in the yml passed as vllm serve --config /opt/stack.yml; patches/ bind-mount over the fork tree, env-gated, they do not change argv vs the yml. corrections_to_card: kv_pool_tokens submitted as the twin card's verbatim 5,280,000 (5.28M) \u2014 repo exact figure 5,282,636 / 5.04x (2026-11-02 gmu 0.95 bump) recorded in performance notes; 195/241/149 c1 = MTP-4 solo code/structured/prose medians (2026-11-02 sparkDash replica); twin card said source_type git \u2014 submission enum maps it to fork (clee/vllm-sm80 q38fn lineage); engine.git_url names the RECIPE repo per card instruction, the engine fork itself is clee/vllm-sm80 (artifacts + install_note); hardware resolves to existing profile nvidia-cmp-170hxx3 (device_count 3). All performance numbers are author-claimed, dated 2026-11-01/02, not validated on our hardware."},"engine_source":{"type":"fork","git_url":"https://github.com/givisok/qwen38-flash-next-170hx-recipe","base_engine":"vllm","pip_package":null,"base_version":"nightly-7d0b4e57 + vllm-sm80 q38fn/stack-local 3c367a58e5","image_digest":null,"install_note":"NOT installable from the stock image: the stack is the clee/qwen3.8-flash-next-3x170hx vllm-sm80 fork (q38fn lineage) on the pinned upstream nightly 7d0b4e57aac4c4323225b0eee9ae960ce75ffffe, with this repo's patches/ series applied; production fork state q38fn/stack-local @ 3c367a58e5 (content = branch dflash2-pool @ 9eadcf0a8f + patches/dflash2-pool-increment.diff), kit main @ f248264. Build vllm-sm80:q38fn from the clee repo's Dockerfile, or bind-mount the fork vllm/ files over the pinned nightly (run-qwen38fn.sh does the bind-mounts); newer nightlies are not validated against this stack. Image is a local build \u2014 no published digest.","pinned_commit":"91803a4a18a1d96d4d73ed1709b11937e6bd1f63"},"credit":[{"name":"recipe, tuning notes and measurements by givisok","url":"https://github.com/givisok/qwen38-flash-next-170hx-recipe"},{"name":"base stack and vllm-sm80 fork by clee","url":"https://github.com/clee/qwen3.8-flash-next-3x170hx"},{"name":"AWQ-g32 checkpoint by leoncca","url":"https://huggingface.co/leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32"},{"name":"uncensored base by orcarouter","url":"https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored"},{"name":"cudagraph capture-size idea from mannydelrio","url":"https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx"},{"name":"vLLM (Apache-2.0)","url":"https://github.com/vllm-project/vllm"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["IMAGE=vllm-sm80:q38fn","FORK=/root/src/vllm-sm80","./run-qwen38fn.sh"],"layouts":["pp"],"runtime":{"env":"NCCL_P2P_DISABLE=1 NCCL_IB_DISABLE=1 NCCL_DEBUG=WARN (P2P hangs on these cards) | VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 (1M > checkpoint native 262144) | VLLM_PP_LAYER_PARTITION=17,17,14 (1 layer off the last rank; rollback 17,16,15) | VLLM_PLE_STAGED=1 VLLM_PLE_MMAP=0 VLLM_PLE_SYNC=1 (sync closes the fp8-PLE H2D staging race) | VLLM_QWEN4_QSA_TOPK_CANONICAL=1 VLLM_QWEN_QSA_TOPK_SORTED=1 (QSA top-k tie fix) | VLLM_QWEN4_ROUTER_FP32=1 VLLM_QWEN4_MTP_PER_STREAM_NORM=0 | VLLM_QWEN4EXP_PP_INT8_HOP=1 (g4bwy 0006 port) | VLLM_MTP_DRAFT_HEAD_PATH=/models/qwen38fn-awq-g32/draft-lm-head-w4a16-g128.safetensors | VLLM_MIXED_PREFILL_CHUNK=864 (FN12 bounded prefill) | VLLM_SWA_BLOCK_CAP=432 VLLM_SWA_NO_PREFIX_CACHE=1 (dflash-lane knobs, inert on the mtp lane) | VLLM_PLE_PREFILL_CHECKPOINT=1 | QWEN4EXP_PLE_MODEL_DIR=/models/qwen38fn-awq-g32 | HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 | PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True PYTORCH_NVML_BASED_CUDA_CHECK=1 VLLM_WORKER_MULTIPROC_METHOD=spawn","notes":"run-qwen38fn.sh (default CONFIG=configs/awq-1m-fp8-ssmbf16-mtp4.yml, SPEC_MODE=mtp, since 2026-11-02) execs docker run -d --gpus all --ipc=host --ulimit memlock=-1 --restart unless-stopped -p 0.0.0.0:8080:8000 with /home/models ro at /models, the config yml ro at /opt/stack.yml, the fork vllm/ tree bind-mounted over the image dist-packages, --entrypoint vllm vllm-sm80:q38fn serve --config /opt/stack.yml. All serve args live in the yml: model /models/qwen38fn-awq-g32, served-model-name qwen3.8, api-key set on server, host 0.0.0.0 port 8000, distributed-executor-backend mp, tensor-parallel-size 1, pipeline-parallel-size 3, disable-custom-all-reduce true, dtype bfloat16, max-model-len 1048576, block-size 1728, max-num-seqs 16, max-num-batched-tokens 8192, long-prefill-token-threshold 2048, gpu-memory-utilization 0.95, kv-cache-dtype fp8_e4m3, enable-prefix-caching, prefix-cache-retention-interval 17280, prefix-match-unit 16, enable-chunked-prefill, mamba-cache-mode align, mamba-ssm-cache-dtype bfloat16, async-scheduling true, compilation-config cudagraph_mode FULL_AND_PIECEWISE cudagraph_capture_sizes [1,2,5,10,15,20,25,30,40,50,60,70,80], enable-flashinfer-autotune false, speculative-config {method mtp, model /models/qwen38fn-awq-g32/mtp-fp8, num_speculative_tokens 4}, chat-template /opt/chat_template.jinja (checkpoint chat_template.jinja), trust-request-chat-template, enable-auto-tool-choice, tool-call-parser qwen3_xml, reasoning-parser qwen3, default-chat-template-kwargs {enable_thinking true, preserve_thinking true, reasoning_effort xhigh (xhigh/medium/low only)}, hf-overrides text_config.layer_types [linear_attention x3, qwen_sparse_attention] x12 and rope_parameters {rope_type yarn, factor 4.0, original_max_position_embeddings 262144, rope_theta 10000000, partial_rotary_factor 0.25, mrope_interleaved true, mrope_section [11,11,10]}. CAVEAT (verbatim from the yml): the disk KV tier (kv-disk) must be wiped when swapping model checkpoints \u2014 entries are content-addressed by tokens, not weights; the kv-transfer-config tiering block itself is REMOVED (fork OffloadingConnector corrupted multi-chunk cached prefixes \u2014 word-salad, ablation 5/5 garbage vs 6/6 clean). Rollback rungs: MTP-3 CONFIG=configs/awq-1m-fp8-ssmbf16.yml; fast code lane valid to depth 262,144 only: SPEC_MODE=dflash (k5+bf16 mamba, code 253-257 solo; drafter has no rope_scaling, speculation dies past the YaRN boundary at ~47 tok/s). Host: Proxmox LXC CT101 Debian, swap 0, 3 cards passed through, memlock unlimited."},"kv_pool_tokens":5280000,"max_context_len":1048576,"storage_requirements":{"note":"leoncca AWQ-g32 checkpoint ~137 GB (87 shards) + mtp-fp8 draft + W4A16 draft head on NVMe; the 51 GB fp8 PLE table streams from NVMe via preadv (never fully loaded).","nvme_gb":140}},"models":[{"slug":"qwen3-8-flash-next-uncensored-awq","name":"Qwen3.8-Flash-Next-Uncensored-AWQ","hf_repo":"leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32","quant":"awq-w4a16-g32-ple-fp8"}],"hardware":[{"slug":"nvidia-cmp-170hxx3","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":3}],"generation":"ampere","tags":["fp8-kv","long-context","mamba-hybrid","mtp-speculative","needle-validated","no-p2p","nvme-staged-ple","pipeline-parallel","sm80","uncensored","yarn-1m"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"All figures author-claimed in the repo at 91803a4a (measured 2026-11-01/02), UNVALIDATED on-box; claimed_by=submitter T0 \u2014 no raw run file ingested, import via POST /api/v1/benchmarks to upgrade. Solo = sparkDash DecodeBench protocol replica (scripts/sd_bench.py), per-request medians, MTP-4 lane: solo c1 variant split code 195 / structured 241 / prose 149 tok/s (one row landed, variants kept here). 1M: scripts/needle_1m.py 2026-11-01 \u2014 1,010,326-token wall, 6 needles x3 phrasings 18/18 PASS at 5-95% depth, cold prefill 235.6 s = 4,287 tok/s, prefix-cache resend TTFT 4.3-4.5 s (~52x). Depth sweep NOTES \u00a721: MTP flat to 1M, 1.63 acc/draft, 134 tok/s @1M. Exact KV pool 5,282,636 tokens / 5.04x after the 2026-11-02 gpu-memory-utilization 0.94->0.95 A/B (runs/gmu095/; 131k battery cold prefill 4,179 tok/s, follow-up-turn TTFT 2.10 s, needle 120k PASS).\", \"points\": [{\"notes\": \"twin-card verbatim: MTP-4 @1M decode 134 tok/s, 1.63 acc/draft (NOTES \u00a721 depth sweep 2026-11-02, no cliff to 1M \u2014 MTP shares target YaRN rope); gate 18/18 multi-needle at 5-95% depth; claimed.\", \"concurrency\": 1, \"context_len\": 1048576, \"decode_tok_s\": 134}, {\"notes\": \"MTP-4 solo 2026-11-02 (sparkDash replica): code 195 / structured 241 / prose 149 tok/s medians; claimed, not validated on-box.\", \"concurrency\": 1, \"context_len\": 4096, \"decode_tok_s\": 195}, {\"notes\": \"MTP-4 N=8 code aggregate 686 tok/s (sparkDash, 2026-11-02); structured N=8 1119; claimed.\", \"concurrency\": 8, \"context_len\": 4096, \"decode_tok_s\": 686}, {\"notes\": \"cold prefill 1,010,326 tokens in 235.6 s = 4,287 tok/s (scripts/needle_1m.py, 2026-11-01); resend TTFT 4.3-4.5 s via prefix cache; 18/18 needle PASS; claimed.\", \"ttft_ms\": 235600, \"context_len\": 1048576, \"prefill_tok_s\": 4287}, {\"notes\": \"131k battery 2026-11-02 at gpu-memory-utilization 0.95: cold prefill 4,179 tok/s, needle 120k PASS (runs/gmu095/); claimed.\", \"context_len\": 131072, \"prefill_tok_s\": 4179}], \"context_len\": 1048576}"}],"perf_points":[{"id":2348,"context_len":4096,"prefill_tok_s":null,"decode_tok_s":195,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"MTP-4 solo 2026-11-02 (sparkDash replica): code 195 / structured 241 / prose 149 tok/s medians; claimed, not validated on-box."},{"id":2349,"context_len":4096,"prefill_tok_s":null,"decode_tok_s":686,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"MTP-4 N=8 code aggregate 686 tok/s (sparkDash, 2026-11-02); structured N=8 1119; claimed."},{"id":2351,"context_len":131072,"prefill_tok_s":4179,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"131k battery 2026-11-02 at gpu-memory-utilization 0.95: cold prefill 4,179 tok/s, needle 120k PASS (runs/gmu095/); claimed."},{"id":2350,"context_len":1048576,"prefill_tok_s":4287,"decode_tok_s":null,"ttft_ms":235600,"concurrency":null,"claimed_by":"submitter","notes":"cold prefill 1,010,326 tokens in 235.6 s = 4,287 tok/s (scripts/needle_1m.py, 2026-11-01); resend TTFT 4.3-4.5 s via prefix cache; 18/18 needle PASS; claimed."},{"id":2347,"context_len":1048576,"prefill_tok_s":null,"decode_tok_s":134,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"twin-card verbatim: MTP-4 @1M decode 134 tok/s, 1.63 acc/draft (NOTES \u00a721 depth sweep 2026-11-02, no cliff to 1M \u2014 MTP shares target YaRN rope); gate 18/18 multi-needle at 5-95% depth; claimed."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-g32-on-3x-cmp-170hx-pp3-1m-ctx.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-g32-on-3x-cmp-170hx-pp3-1m-ctx.json"},"requirements":null,"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/givisok/qwen38-flash-next-170hx-recipe","pin":"91803a4a18a1d96d4d73ed1709b11937e6bd1f63 (HEAD 2026-11-02) \u2014 run-qwen38fn.sh + configs/awq-1m-fp8-ssmbf16-mtp4.yml + patches/ are the verbatim argv/env source"},{"kind":"engine_image","pointer_type":"git_commit","pointer":"https://github.com/clee/vllm-sm80","pin":"q38fn lineage on upstream nightly 7d0b4e57aac4c4323225b0eee9ae960ce75ffffe; production fork tree q38fn/stack-local @ 3c367a58e5 (= dflash2-pool @ 9eadcf0a8f + increment diff); kit main @ f248264; image vllm-sm80:q38fn = local build, no published digest"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/leoncca/Qwen3.8-Flash-Next-Uncensored-AWQ-g32","pin":"AWQ W4A16 g32 (~137 GB, 87 shards, damped GPTQ 169K rows, KL 0.0214 / top-1 97.08% vs BF16); fp8 PLE table 51 GB; calibrated QSA KV scales; mtp-fp8 draft included; verify SHA256SUMS"},{"kind":"drafter","pointer_type":"hf_model","pointer":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2","pin":"DFlash2 drafter (Apache-2.0, 1.1 GB) for the SPEC_MODE=dflash fast code lane \u2014 speculation only valid to depth 262,144 (drafter carries no rope_scaling)"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-g32-on-3x-cmp-170hx-pp3-1m-ctx.json","alternate":"/r/qwen3-8-flash-next-uncensored-awq-g32-on-3x-cmp-170hx-pp3-1m-ctx","describedby":"/openapi.json"},"updated_at":"2026-11-02 03:44:32.301706-07","created_at":"2026-11-02 03:44:32.301706-07","speeds":{"decode_tok_s":164.5,"prefill_tok_s":4233,"leaderboard":"/api/v1/leaderboard.json"}}