{"slug":"qwen3-8-flash-next-swift15-awq-2x-cmp-170hx-pp2-mtp4-ple-asgard","title":"Qwen3.8-Flash-Next Swift15 AWQ 2x CMP 170HX PP2 MTP4 PLE Asgard","summary":"Qwen3.8-Flash-Next (Swift 1.5 W4A16-AWQ, DotH MTP head retrained on agent traffic) on two unlocked 64 GB CMP 170HX cards (300 W VBIOS cross-flash) in vLLM pipeline parallel. PP2 (24,24 layers), MTP4 with n-gram lookup and an int4 draft lm_head, --prefix-match-unit 16, BF16 KV, the ~95 GiB PLE n-gram table streamed from NVMe via Linux AIO (16 GiB row cache), 262,144-token max-model-len, 24 seqs, Qwen Asgard chat template with reasoning_effort medium. Author-measured (night 6, 2026-11-05): 199.6 tok/s median code-gen decode at default clocks with 3.62 mean acceptance, 304 tok/s code editing at 1440 MHz, 8,403 tok/s prefill on a 128K prompt, 511 tok/s aggregate at 16 users, BFCL v4 80.3% (night-3 suite; night-6 re-run 80.0%). recipe by inordic.","license":"MIT","tier":"community","beta":false,"revision":{"no":1,"commit_sha":"0d29eff29115fd16cf2916943918d514b8ed9f24","content_hash":"0e59c52a01761130291f76a561d817d3531f8bab","created_at":"2026-11-04 21:50:48.787451-07","notes":"Ingested from the lar-scout 2026-11-05 sweep; re-verified at repo HEAD fe725e47 (night-6 production profile dated 2026-11-05). corrections_to_card: (1) PLE table is ~95 GiB streamed from NVMe via Linux AIO with a 16 GiB row cache, not '48 GiB mmapped' (mmap is only the >=93 GB RAM fallback); (2) aggregate at 16 users on the night-6 standard is 511.0 tok/s (night6-conc.jsonl C-u16) - the card's 504 is the night-2 previous standard; (3) BFCL 80.3%/HumanEval+ 95.7%/MBPP+ 82.5%/LCB 58.3%/needles are the nights 2-3 suite (night-6 re-run BFCL 80.0%, within noise); (4) code-edit 304 tok/s measured at 1440 MHz on the night-4/5 standard (night-6 re-check 304.3); (5) credit list corrected to the repo's NOTICE.md - the card's 'Morrowmake' and 'g4bwy' credits appear nowhere in the repo (patches 19-23 are ports of vllm#54912 #62344 #65470; K1/D3 patches are inordic's own); (6) patch 14 (C3) is not redistributed in the public repo and is OFF. Dedupe: 'inordic' 0 hits across all served recipe bodies/corpus; distinct from the other 2x-170HX W4A16 slugs by checkpoint (ukisai Swift 1.5 AWQ + DotH MTP head), engine tree (n6k), template (Asgard), int4 draft head, --prefix-match-unit 16."},"engine_source":{"type":"fork","git_url":"https://github.com/inordic/cmp170hx-vllm-qwen38-flash-next","base_engine":"vllm","pip_package":null,"base_version":"0.29.1rc1.dev402+ga5a30471f (n6k tree, patches 01-23)","image_digest":null,"install_note":"Not a pip package: a venv at /opt/vllm-qfn-n6k built by patches/make-venv.sh. Base = upstream vLLM a5a30471ff2bb7f0824f2da10e358af98d304472 (0.29.1rc1.dev402) + iIIusi0n's PLE-SSD recipe (ple_ssd_io.so helper) + 2dameneko's build at commit 56e1347, then this repo's patches 01-18 and night-6 set 19-23 (terse alias, QSA ring, QSA indexer sm80 kernels; ports of vllm#55601 #60210 #56734 #60615 #58784 #54912 #62344 #65470). Launch goes through 2dameneko's serve/serve.sh with RUN_ENV=config/qfn-prod.env (systemd unit config/vllm-qfn.service); the argv here is serve.sh's exec line assembled from that profile. Required artifacts: optimization/ple_ssd_io.so, templates/qwen-asgard-v1.jinja (leonbollerup/qwen-asgard-jinja), models/draft-heads/swift15-awq-lm_head-w4a16.safetensors (tools/models/quantize_lm_head.py), and the DotH MTP head grafted from doth4580/Qwen3.8-Flash-Next-DotH-MTP-BF16 (tools/models/doth.md). Abliteration patch 14 is not redistributed and is OFF. CUDA arch sm80 (GA100).","pinned_commit":"fe725e47ec81de2968cdc2df5cf14ef40942f055"},"credit":[{"name":"inordic (recipe, patch series, measurements)"},{"name":"2dameneko (base vLLM build vllm-Qwen3.8-Flash-Next-2xCMP170HX, commit 56e1347)"},{"name":"iIIusi0n (PLE-SSD offload recipe and ple_ssd_io.so helper)"},{"name":"ukisai (Swift 1.5 W4A16-AWQ checkpoint)"},{"name":"doth4580 (DotH MTP head retrained on agent traffic)"},{"name":"leonbollerup (Qwen Asgard chat template)"},{"name":"AlexBrandes (int4 MTP draft head patch 08, quantize_lm_head.py)"},{"name":"vLLM upstream PRs ported: #55601, #60210, #56734, #60615, #58784, #54912, #62344, #65470"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"env":["VLLM_VENV=/opt/vllm-qfn-n6k/.venv","MODEL=/opt/vllm-qfn/models/swift15-doth","SERVED_NAME=qwen3.8-flash-next","PORT=8080","CUDA_DEVICES=1,0","MTP=0","PP_SIZE=2","VLLM_PP_LAYER_PARTITION=24,24","MAX_MODEL_LEN=262144","MAX_NUM_SEQS=24","MAX_BATCH_TOKENS=4096","GPU_MEMORY=0.90","PLE_SSD_OFFLOAD=1","PLE_SSD_WORKERS=32","PLE_SSD_CACHE_MB=16384","PLE_SSD_IO_DEPTH=512","PLE_SSD_PREFETCH=131072","PLE_SSD_LIB=/opt/vllm-qfn/optimization/ple_ssd_io.so","COMPILE_MODE=1","GRAPH_MODE=FULL_AND_PIECEWISE","VLLM_USE_BREAKABLE_CUDAGRAPH=1","VLLM_PLE_NONPLE_LOADER=instanttensor","QWEN38_ABLITERATION=","CAPTURE_SIZES=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,18,20,22,24,26,28,30,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,128,192,256,512,1024,2048]","VLLM_MTP_DRAFT_HEAD_PATH=/opt/vllm-qfn/models/draft-heads/swift15-awq-lm_head-w4a16.safetensors"],"launch":["/opt/vllm-qfn-n6k/.venv/bin/vllm","serve","/opt/vllm-qfn/models/swift15-doth","--served-model-name","qwen3.8-flash-next","--host","0.0.0.0","--port","8080","--dtype","bfloat16","--max-model-len","262144","--max-num-seqs","24","--max-num-batched-tokens","4096","--gpu-memory-utilization","0.90","--safetensors-load-strategy","lazy","--mamba-cache-mode","align","--enable-prefix-caching","--enable-chunked-prefill","--reasoning-parser","qwen3","--enable-auto-tool-choice","--tool-call-parser","qwen3_xml","--additional-config","{\"ple_ssd_offload\":true,\"ple_ssd_workers\":32,\"ple_ssd_cache_mb\":16384,\"ple_ssd_native_library\":\"/opt/vllm-qfn/optimization/ple_ssd_io.so\",\"ple_ssd_io_depth\":512,\"ple_ssd_prefetch_tokens\":131072}","--pipeline-parallel-size","2","--enable-expert-parallel","--async-scheduling","--prefix-match-unit","16","--enable-mamba-fine-grained-prefix-cache","--served-model-name","qwen3.8-flash-next","qwen3.8-flash-next-terse","swift-1.5-awq","Qwen3.8-Flash-Next-AWQ-INT4","swift-1.5-iq3_xxs","--chat-template","/opt/vllm-qfn/templates/qwen-asgard-v1.jinja","--default-chat-template-kwargs","{\"reasoning_effort\":\"medium\",\"tool_prompt\":\"chromix\"}","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":4,\"ngram_lookup\":true}","--compilation-config","{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,18,20,22,24,26,28,30,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,128,192,256,512,1024,2048],\"mode\":1}","--api-key","<from /root/llm-api-key>"],"layouts":["pp"],"runtime":{"notes":"Card order matters: CUDA_DEVICES=1,0 puts the slower (250 W VBIOS) card as PP stage 0: +9% decode, prefill equal. --prefix-match-unit 16 because with 4 draft tokens the attention block is 1616 tokens, which 32 does not divide. reasoning_effort medium is the measured optimum (xhigh costs -5.2 BFCL points). Production profile config/qfn-prod.env (night 6, 2026-11-05) is the argv authority; systemd unit config/vllm-qfn.service. INT8 KV (patch 10/K1) is validated but NOT in production (BF16 KV kept)."},"kv_pool_tokens":838024,"max_context_len":262144,"storage_requirements":{"note":"~95 GiB BF16 PLE n-gram table streamed from NVMe (Linux AIO, 16 GiB row cache); 166 GB checkpoint also on NVMe; container RAM only 55 GB.","nvme_gb":95}},"models":[{"slug":"qwen3-8-flash-next-w4a16-awq-swift15","name":"Qwen3.8-Flash-Next-W4A16-AWQ-Swift15","hf_repo":"ukisai/Swift-1.5-Qwen3.8-Flash-Next-W4A16-AWQ","quant":"w4a16-awq-int4-draft-lm-head"}],"hardware":[{"slug":"cmp-170hx-x2","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":2}],"generation":"ampere","tags":["asgard-template","bf16-kv","long-context","mtp4","no-p2p","ple-nvme","pp2","prefix-caching","sm80","speculative-decoding"],"tasks":["chat","code"],"aa":{"aa_index":40,"aa_provenance":"measured","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-flash-next","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-06 18:30:00-06","aa_family_slug":"qwen3-8-flash-next","aa_resolved_slug":"qwen3-8-flash-next"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Author-measured, single box (i7-8700K, 64 GB RAM, NVMe, unprivileged LXC, PCIe Gen2 x8, no P2P). Production checks at default clocks (cards hold ~1455-1485 MHz under load); A/B rows locked at 1440 MHz. Decode 199.6 = median of decode-N6-prod.json (9,216-token code-gen prompt, T=0.6, acceptance 3.62, results/logs/night6-NIGHT-LOG.md 01:03:55 UTC). Code-edit 304.0 at 1440 MHz (night-4/5 standard edit-D4p16.txt/edit-R.txt; night-6 re-check 304.3); not yet re-measured on the night-6 standard at default clocks. Concurrency rows from results/concurrency/night6-conc.jsonl (final build C, 1K prompts, 150 s, 0 errors; aggregate output tok/s). context_len on the code-edit row is the tuple marker 9216+10240. Quality: BFCL v4 80.3% (2,080 items), HumanEval+ 95.7%, MBPP+ 82.5%, LiveCodeBench v6 58.3%, long-context needles 32K/64K/110K = 100/96.7/95.0% - nights 2-3 suite (results/quality/); night-6 re-run BFCL 80.0% within noise. KV pool 838,024 tokens (night-6 startup log).\", \"tok_s\": 199.599999999999994315658113919198513031005859375, \"points\": [{\"notes\": \"Code-generation decode median at default clocks, T=0.6, MTP4+ngram+int4 draft head, mean acceptance length 3.62 (decode-N6-prod.json).\", \"concurrency\": 1, \"context_len\": 9216, \"decode_tok_s\": 199.599999999999994315658113919198513031005859375}, {\"notes\": \"Code-editing decode, thinking off, clocks locked 1440 MHz (edit-D4p16/edit-R; night-6 re-check 304.3). context_len is the tuple marker 9216+10240 (edit bench prompt size unstated).\", \"concurrency\": 1, \"context_len\": 19456, \"decode_tok_s\": 304}, {\"notes\": \"Aggregate output tok/s, 16 users, 1K prompts, 150 s, 0 errors (night6-conc.jsonl C-u16, 1440 MHz); TTFT median 0.43-0.45 s.\", \"concurrency\": 16, \"context_len\": 1024, \"decode_tok_s\": 511}, {\"notes\": \"Aggregate output tok/s, 8 users, 1K prompts, 150 s, 0 errors (night6-conc.jsonl C-u8, 1440 MHz).\", \"concurrency\": 8, \"context_len\": 1024, \"decode_tok_s\": 400.19999999999998863131622783839702606201171875}, {\"notes\": \"Cold 128K real-text prompt prefill, 1440 MHz (exact-n6-K.json; night-6 kernels +2.9% over the prior standard).\", \"context_len\": 131072, \"prefill_tok_s\": 8403}, {\"notes\": \"Cold 32K real-text prompt prefill, 1440 MHz (night-6 after row).\", \"context_len\": 32768, \"prefill_tok_s\": 7847}], \"context_len\": 9216}"}],"perf_points":[{"id":2430,"context_len":1024,"prefill_tok_s":null,"decode_tok_s":400.19999999999998863131622783839702606201171875,"ttft_ms":null,"concurrency":8,"claimed_by":"submitter","notes":"Aggregate output tok/s, 8 users, 1K prompts, 150 s, 0 errors (night6-conc.jsonl C-u8, 1440 MHz)."},{"id":2429,"context_len":1024,"prefill_tok_s":null,"decode_tok_s":511,"ttft_ms":null,"concurrency":16,"claimed_by":"submitter","notes":"Aggregate output tok/s, 16 users, 1K prompts, 150 s, 0 errors (night6-conc.jsonl C-u16, 1440 MHz); TTFT median 0.43-0.45 s."},{"id":2427,"context_len":9216,"prefill_tok_s":null,"decode_tok_s":199.599999999999994315658113919198513031005859375,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Code-generation decode median at default clocks, T=0.6, MTP4+ngram+int4 draft head, mean acceptance length 3.62 (decode-N6-prod.json)."},{"id":2428,"context_len":19456,"prefill_tok_s":null,"decode_tok_s":304,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Code-editing decode, thinking off, clocks locked 1440 MHz (edit-D4p16/edit-R; night-6 re-check 304.3). context_len is the tuple marker 9216+10240 (edit bench prompt size unstated)."},{"id":2432,"context_len":32768,"prefill_tok_s":7847,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"Cold 32K real-text prompt prefill, 1440 MHz (night-6 after row)."},{"id":2431,"context_len":131072,"prefill_tok_s":8403,"decode_tok_s":null,"ttft_ms":null,"concurrency":null,"claimed_by":"submitter","notes":"Cold 128K real-text prompt prefill, 1440 MHz (exact-n6-K.json; night-6 kernels +2.9% over the prior standard)."}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-swift15-awq-2x-cmp-170hx-pp2-mtp4-ple-asgard.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-swift15-awq-2x-cmp-170hx-pp2-mtp4-ple-asgard.json"},"requirements":null,"artifacts":[{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/inordic/cmp170hx-vllm-qwen38-flash-next","pin":"fe725e47ec81de2968cdc2df5cf14ef40942f055"},{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-Flash-Next-W4A16-AWQ","pin":"598afc96128f76a67117caf878f538d34f6f1e02"},{"kind":"drafter","pointer_type":"url","pointer":"https://huggingface.co/doth4580/Qwen3.8-Flash-Next-DotH-MTP-BF16","pin":"grafted dense MTP head via tools/models/graft_doth.py; int4 draft lm_head built by tools/models/quantize_lm_head.py"}],"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-swift15-awq-2x-cmp-170hx-pp2-mtp4-ple-asgard.json","alternate":"/r/qwen3-8-flash-next-swift15-awq-2x-cmp-170hx-pp2-mtp4-ple-asgard","describedby":"/openapi.json"},"updated_at":"2026-11-04 21:50:48.787451-07","created_at":"2026-11-04 21:50:48.787451-07","speeds":{"decode_tok_s":251.80000000000001136868377216160297393798828125,"prefill_tok_s":8125,"leaderboard":"/api/v1/leaderboard.json"}}