{"slug":"qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix","title":"Qwen3.8-27B INT8 MTP3 prefix-cache on 1x CMP 170HX Hynix","summary":"Qwen3.8-27B at 65-67 tok/s decode (74 structured) with MTP k=3 speculative decoding AND prefix caching simultaneously on a single NVIDIA CMP 170HX 64GB (Hynix HBM2e, GA100/sm80), 262,144-token context on fp8_e4m3 KV, 24/7 production on pinned upstream vLLM nightly 0.26.1rc1.dev1024+g1fe3a1571 \u2014 the first build where prefix caching and MTP spec decode coexist (GDN mamba-align). Cached TTFT on a 7K shared prefix drops ~4.0 s -> 1.44 s (33.5% hit rate) while decode gains 7-9% over the 0.27.1 fork fallback \u2014 the 'MTP vs prefix cache, pick one' tradeoff is gone. Load-bearing trio --async-scheduling + --max-num-batched-tokens 8192 + --max-num-scheduled-tokens 8192 (default chunked-prefill batch = 0.0% hit rate while looking healthy); --prefix-match-unit 400 halves cached TTFT further (must divide the 1600-token GDN block); batch budget 32000 OOMs and hard-wedges the card. Samsung-40GB side recipe (int8 + DFlash2, 97.8 tok/s) with an sm80 int8-repack CPU-fallback patch.","license":"CC-BY-4.0","tier":"community","beta":false,"revision":{"no":2,"commit_sha":"ad3f55dab02cedfb2b187f7dcfc3f1647e0aaa0c","content_hash":"68f95048b88c132d1ddbffc8ceceaf5c38be5140","created_at":"2026-10-20 15:34:22.585629-06","notes":"Submitted 2026-10-09 by localai-recipes from lar-scout ingest card t_35407999 (sweep QUEUE #3, 2026-10-09). Repo ahnguyen17/cmp-170hx-vllm @ 192cc330d9434e915bab05a2263944ae5fd12361 (HEAD via git ls-remote 2026-10-09; created 2026-08-20, 11 commits, actively maintained). Launch argv token-verified verbatim against serve-qwen38-nightly.sh at that sha; flag rationale cross-checked against script comments + README flag-by-flag table + docs/gdn-prefix-caching-2026-08-23.md; performance figures carried from the README A/B table and deep-dive probes as submitter-T0 with in-repo receipts (same-day A/B, 3-run medians, usage-counted). Engine pins: upstream nightly wheel vllm==0.26.1rc1.dev1024 from wheels.vllm.ai/nightly (commit 1fe3a1571ac67581478a11743e55a306de1d136f), torch 2.13.0+cu130, driver 610.43.03; model HF rev 7c12373712d1363e2b76655cb3332c9c124627d7. FIRST single-170HX recipe on the site (buckets were x2/x3/x4 only, verified live 2026-10-09); first GPTQ-int8/W8A16 lane on any 170HX recipe (site's other 170HX entries are W4A16/FP8/EXL3); first upstream-nightly vLLM 170HX entry (existing ones use forks). Documents the Hynix-vs-Samsung memory-bin split with working recipes for both. Hardware resolves to the pre-existing cmp-170hx single-card profile via the 0041 seed-slug convention (no orphan bucket)."},"engine_source":{"type":"release","git_url":null,"base_engine":"vllm","pip_package":"vllm==0.26.1rc1.dev1024","base_version":"0.26.1rc1.dev1024+g1fe3a1571 (upstream nightly wheel)","image_digest":null,"install_note":"Install from the upstream nightly wheel index: uv pip install --extra-index-url https://wheels.vllm.ai/nightly vllm==0.26.1rc1.dev1024 (commit-pinned build; nightly is a moving target \u2014 pin exactly this). Validated on torch 2.13.0+cu130, driver 610.43.03. No patches needed on this build (the GDN rework supersedes vllm-project/vllm#50021); the repo's patches/vllm-pr50021-gdn-spec-bounds.patch is required ONLY for the vLLM 0.27.1 fallback recipe (serve-qwen38-mtp.sh) \u2014 without it, unbounded GDN state reads under MTP spec decode crash the server.","pinned_commit":"1fe3a1571ac67581478a11743e55a306de1d136f"},"credit":[{"name":"ahnguyen17 (recipe, patches, probes, A/B benchmarks)","url":"https://github.com/ahnguyen17/cmp-170hx-vllm"},{"name":"lued (Qwen3.8-27B-INT8-W8A16-MTP checkpoint)","url":"https://huggingface.co/lued/Qwen3.8-27B-INT8-W8A16-MTP"},{"name":"vllm-project (vLLM upstream; #50021 GDN spec bounds, #45238 align-mode cache retention)","url":"https://github.com/vllm-project/vllm"}],"author":{"name":"localai-recipes","isSite":false},"uploaded_by":"localai-recipes","config":{"launch":["vllm","serve","/path/to/Qwen3.8-27B-INT8-W8A16-MTP","--quantization","gptq","--dtype","bfloat16","--api-key","$API_KEY","--max-model-len","262144","--gpu-memory-utilization","0.90","--kv-cache-dtype","fp8_e4m3","--mamba-ssm-cache-dtype","auto","--enable-prefix-caching","--enable-chunked-prefill","--prefix-match-unit","400","--async-scheduling","--max-num-batched-tokens","8192","--max-num-scheduled-tokens","8192","--speculative-config","{\"method\":\"mtp\",\"num_speculative_tokens\":3,\"draft_sample_method\":\"probabilistic\"}","--compilation-config","{\"max_cudagraph_capture_size\":128,\"custom_ops\":[\"+rms_norm\",\"+silu_and_mul\"]}","--enable-auto-tool-choice","--tool-call-parser","qwen3_coder","--reasoning-parser","qwen3","--served-model-name","qwen3.8-27b","--host","0.0.0.0","--port","8012"],"runtime":{"env":"PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (prevents fragmentation OOM on the 64GB HBM2e across long uptimes \u2014 Samsung sm80 bins instead wedge with it and want :False); CUDA_DEVICE_ORDER=PCI_BUS_ID + CUDA_VISIBLE_DEVICES pinned (FASTEST_FIRST hands the server the wrong GPU in mixed rigs); VLLM_USE_FLASHINFER_SAMPLER=1 (verbatim serve-qwen38-nightly.sh @ 192cc33).","notes":"THE CRITICAL FLAG TRIO (verbatim script comments): --async-scheduling + --max-num-batched-tokens 8192 + --max-num-scheduled-tokens 8192 \u2014 with the default chunked-prefill batch size the prefix cache silently runs at 0.0% hit rate while everything looks healthy: the mamba-align split logic needs single-shot prefills that end exactly on the 1600-token GDN hash boundary to register cacheable blocks. Do NOT raise max-num-batched-tokens above 8192: GDN prefill intermediates scale with scheduled tokens \u2014 32000 OOM'd in chunk_fwd_o and hard-wedged the GA100 card (reboot-only recovery). --prefix-match-unit 400 must divide the 1600-token GDN block (512 boot-fails with a ValueError). MTP k>3 is a measured dead end (acceptance saturates ~3.2/8 at k=7; k=4 and k=7 both net -27..45% tok/s). --enable-prefix-caching auto-enables mamba-cache-mode align. Samsung-memory 40GB 170HX fallback inside the repo (serve-g3-int8df2.sh, port 18020): int8 + DFlash2 via the qwen38-27b-rtx3090 fork (SPEC=dflash2 CTX=fast), 97.8 tok/s single-stream / 275 tok/s aggregate x8, with patches/sm80-int8-repack-cpu-fallback.patch (Xid-31 wedge during gptq_marlin_repack at production occupancy; CPU-side repack is bit-exact, ~3 extra boot minutes) and expandable_segments:False \u2014 opposite allocator setting from the Hynix recipe; Samsung bins still throw Xid-79 under sustained load."},"max_context_len":262144},"models":[{"slug":"qwen3-8-27b-int8-w8a16-mtp","name":"Qwen3.8-27B-INT8-W8A16-MTP","hf_repo":"lued/Qwen3.8-27B-INT8-W8A16-MTP","quant":"int8-w8a16"}],"hardware":[{"slug":"nvidia-cmp-170hx","vendor":"nvidia","card":"cmp-170hx","card_family":"170hx","vram_gb":64,"device_count":1}],"generation":"ampere","tags":["cuda-graphs","fp8-kv","gdn-fusion","int8","long-context","mtp3","mtp-speculative","prefix-cache","single-gpu","spec-decode"],"tasks":["chat","code"],"aa":{"aa_index":34,"aa_provenance":"published","aa_source_url":"https://artificialanalysis.ai/models/qwen3-8-27b","aa_index_version":null,"aa_methodology_version":"v4.3.2","aa_published_fetched_at":"2026-10-07 18:00:00-06","aa_family_slug":"qwen-3-8-27b","aa_resolved_slug":"qwen-3-8-27b"},"performance":[{"claimed_by":"submitter","metrics":"{\"notes\": \"Same-day A/B nightly-vs-0.27.1-fork on identical harness + hardware (300-token single-stream gens, 3-run medians, tokens counted via usage.completion_tokens \u2014 SSE delta-counting reads a false ~30 tok/s because MTP emits ~3.5 tok/delta), from README.md @ repo HEAD: decode prose greedy 65.4 vs 60.0 tok/s, prose temp 0.7 66.8 vs 62.5, structured greedy 74.0 vs 74.1; MTP acceptance 3.02 vs 3.12 of 4; TTFT on a 7K shared prefix: cold ~6.7 s nightly / ~4.0 s fork (no cache), CACHED 1.44 s at 33.5% hit rate vs ~4.0 s flat on the fork (which cannot combine prefix caching with spec decode). Deep-dive doc: decode flat 69-79 tok/s 6K->40K ctx (79.0 @6K, 68.7 @40K usage-counted); cold prefill collapses superlinearly (17.3k tok/s @6K -> 0.5k @80K, Triton GDN chunk kernels); turn-2 cache miss root-caused to upstream vllm#45238 (align-mode single-checkpoint retention, ~3 sightings to hit), mitigated not eliminated by --prefix-match-unit 400 (20K turn-3 1.67->0.70 s, 40K 5.56->2.79 s).\", \"tok_s\": 65.400000000000005684341886080801486968994140625, \"points\": [{\"notes\": \"Prose greedy, 300-token gens, 3-run median \u2014 README same-day A/B: 65.4 tok/s nightly vs 60.0 on the 0.27.1 fork (+9%)\", \"concurrency\": 1, \"context_len\": 7200, \"decode_tok_s\": 65.400000000000005684341886080801486968994140625}, {\"notes\": \"Prose temp 0.7, 3-run median \u2014 66.8 tok/s nightly vs 62.5 fork (+7%)\", \"concurrency\": 1, \"context_len\": 7200, \"decode_tok_s\": 66.7999999999999971578290569595992565155029296875}, {\"notes\": \"Structured output greedy, 3-run median \u2014 74.0 tok/s nightly vs 74.1 fork (parity)\", \"concurrency\": 1, \"context_len\": 7200, \"decode_tok_s\": 74}, {\"notes\": \"TTFT on a 7K shared prefix, prefix cache HIT (33.5% hit rate) \u2014 1.44 s vs ~4.0 s flat on the 0.27.1 fork which cannot combine prefix caching with MTP spec decode; cold ~6.7 s\", \"ttft_ms\": 1440, \"concurrency\": 1, \"context_len\": 7000}, {\"notes\": \"Decode deep in context @6K via usage.completion_tokens (docs/gdn-prefix-caching-2026-08-23.md: 256 tok / 3.24 s); delta-counting probes read a false ~30 tok/s\", \"concurrency\": 1, \"context_len\": 6000, \"decode_tok_s\": 79}, {\"notes\": \"Decode deep in context @40K, usage-counted \u2014 flat decode band 69-79 tok/s across 6K-40K\", \"concurrency\": 1, \"context_len\": 40000, \"decode_tok_s\": 68.7000000000000028421709430404007434844970703125}], \"context_len\": 262144}"}],"perf_points":[{"id":1171,"context_len":6000,"prefill_tok_s":null,"decode_tok_s":79,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Decode deep in context @6K via usage.completion_tokens (docs/gdn-prefix-caching-2026-08-23.md: 256 tok / 3.24 s); delta-counting probes read a false ~30 tok/s"},{"id":1172,"context_len":7000,"prefill_tok_s":null,"decode_tok_s":null,"ttft_ms":1440,"concurrency":1,"claimed_by":"submitter","notes":"TTFT on a 7K shared prefix, prefix cache HIT (33.5% hit rate) \u2014 1.44 s vs ~4.0 s flat on the 0.27.1 fork which cannot combine prefix caching with MTP spec decode; cold ~6.7 s"},{"id":1173,"context_len":7200,"prefill_tok_s":null,"decode_tok_s":65.400000000000005684341886080801486968994140625,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Prose greedy, 300-token gens, 3-run median \u2014 README same-day A/B: 65.4 tok/s nightly vs 60.0 on the 0.27.1 fork (+9%)"},{"id":1174,"context_len":40000,"prefill_tok_s":null,"decode_tok_s":68.7000000000000028421709430404007434844970703125,"ttft_ms":null,"concurrency":1,"claimed_by":"submitter","notes":"Decode deep in context @40K, usage-counted \u2014 flat decode band 69-79 tok/s across 6K-40K"}],"agent":{"prompt":"Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.","curl":"curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json"},"requirements":null,"artifacts":[{"kind":"model_weights","pointer_type":"hf_model","pointer":"https://huggingface.co/lued/Qwen3.8-27B-INT8-W8A16-MTP","pin":"7c12373712d1363e2b76655cb3332c9c124627d7 \u2014 compressed-tensors INT8 W8A16 (symmetric, g128) with byte-preserved BF16 MTP head; base Qwen/Qwen3.8-27B"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/ahnguyen17/cmp-170hx-vllm","pin":"192cc330d9434e915bab05a2263944ae5fd12361 @ HEAD (git ls-remote 2026-10-09) \u2014 serve-qwen38-nightly.sh (verbatim launch argv), serve-qwen38-mtp.sh (0.27.1 fallback), docs/gdn-prefix-caching-2026-08-23.md, probes/, patches/"},{"kind":"reference","pointer_type":"git_commit","pointer":"https://github.com/vllm-project/vllm","pin":"1fe3a1571ac67581478a11743e55a306de1d136f \u2014 upstream nightly commit behind the 0.26.1rc1.dev1024 wheel; references vllm#50021 (GDN spec bounds, 0.27.1-only) and #45238 (align-mode turn-2 miss)"}],"links":{"self":"/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json","alternate":"/r/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix","describedby":"/openapi.json"},"updated_at":"2026-10-20 15:34:22.585629-06","created_at":"2026-10-08 20:56:47.17406-06","speeds":{"decode_tok_s":68.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"leaderboard":"/api/v1/leaderboard.json"}}