{"meta":{"count":21,"page":1,"pages":2,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["170hx-x2","rtx-pro-6000-blackwell"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T16:24:48+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3","title":"Qwen3.8-Flash-Next NVFP4 on RTX PRO 6000 Blackwell \u2014 TP1 MTP3","summary":"Single-card Blackwell recipe for Qwen3.8-Flash-Next (hybrid GDN/QSA MoE, 48 layers, 512 experts) on one 96 GB RTX PRO 6000: the Community image community image R34 (localinferencelab/vllm, v0.26.1rc0+glm53.r34), TP1 with MTP3 speculative decoding (b12x MoE backend), 256k context. Benchmark-qualified on RTX PRO 6000 Blackwell Max-Q: decode 148 tok/s at ctx-0 (temp 1), prefill 3860 tok/s at 2048 batched tokens. Launch argv transcribed verbatim from the r35 community deployment. Weights: JMNI-Labs/Qwen3.8-Flash-Next-NVFP4-QAD5500-Hybrid \u2014 step-5500 QAD trained tensors recombined with main 7c4f1bc1 MXFP8 attention + NVFP4 MTP experts; same-stack measured +5% MTP-normalized decode, +25% with MTP vs the main checkpoint (evals within noise, MMLU-Pro 66.9 vs 67.2).","tier":"curated","beta":false,"generation":"blackwell","decode_tok_s":137.19999999999998863131622783839702606201171875,"prefill_tok_s":6627.3000000000001818989403545856475830078125,"updated_at":"2026-10-29 23:29:38.177885-06","commit_sha":"28709ecba026aff18371426bc26025e5a4185209","content_hash":"1e37b0f978392ba766c01cfdf9ae9ee3e002ee84","votes":1,"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/rtx6kpro"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json","alternate":"/r/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3"}},{"slug":"glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash max effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records max reasoning effort, selected per request with model GLM-5.3-Flash-max. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":41.5,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:37.835428-06","commit_sha":"663174ab27ccbc0f93296953b897b19e08b64726","content_hash":"1bf0157fd231d61ac68175f7a1493df8e9583746","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash low effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records low reasoning effort, selected per request with model GLM-5.3-Flash-low. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":41.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:37.671135-06","commit_sha":"7a55561ae92fcdeb4a011950b04c4848b7666861","content_hash":"1c7b1a323fae222fd633e500b576c9f1fe1e7cec","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash high effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records high reasoning effort, selected per request with model GLM-5.3-Flash-high. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":43.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:26.881493-06","commit_sha":"26ce83d34683309b83e4478edb3ed1213018d87f","content_hash":"58f06fe95f6c37780466d207181753f0f7c5b12f","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3","title":"Qwen3.8-Flash-Next W4A16 INT4PLE on 2x modded CMP 170HX \u2014 PP2 MTP3 (DUFRelic)","summary":"Production 2x CMP 170HX recipe for Qwen3.8-Flash-Next: zankich INT4PLE repackage of halt95 Merlin W4A16 (94.4 GiB, 51B-param PLE table repacked INT4) on stock vLLM 0.30.0 + a 6-layer hash-guarded overlay stack (PP2/PLE #56444, KV filter #54793, MTP-over-PP #54709, slot statefix #55506, SpeedSuite 5-PR overlay, batched-P2P). PP2/TP1, MTP 3-token draft (measured 52.27% accept). Hardware is the recipe's own mod stack: 74-SM unlock (cmpunlocker), VBIOS 92.00.6D.00.0A, 170tune 1420 MHz SM cap + HBM 1728 MHz undervolt, BAR1 P2P driver patches, 140 W/card. Author-measured 2026-10-03 (median of 3, isolation audit): prefill C1 6213.7 tok/s @16k, decode C1 133.4 tok/s (SD 22.8 reported), decode C4 agg 245.2 tok/s; 55/56 C at 160 W peaks, no Xid/OOM. INT4 PLE table (~26.8 GiB) pinned in HOST RAM; KV BF16 (pinned build's QSA path rejects FP8). Driver mods optional for function but required for the published throughput.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":133.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:28.370851-06","commit_sha":"c26a7fd62ded418a43f5a10abf82eaedb7afc6bc","content_hash":"7c6051a80e2a07b8bca384eb300935641c88349b","votes":0,"credit":[{"name":"DUFRelic","url":"https://github.com/DUFRelic/qwen38-flash-next-recipe"},{"name":"zankich (INT4-PLE repackage + plugin)","url":"https://github.com/zankich/vllm"},{"name":"halt95 (W4A16-Merlin base checkpoint)","url":"https://github.com/halt95/merlin-engine"},{"name":"michael-lgtm (public overlay rebuild)","url":"https://github.com/michael-lgtm/qwen38-flash-next-cmp-overlay"},{"name":"amoghmunikote (cmpunlocker SM unlock + BAR1 P2P)","url":"https://github.com/amoghmunikote/cmpunlocker"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3.json","alternate":"/r/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3"}},{"slug":"qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6","title":"Qwen3.8-Flash-Next NVFP4 2x CMP 170HX TEP2 GDS SSD PLE MTP6","summary":"Recipe by nguyenthimy2022kg-alt. Community patched vLLM run of the NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on 2x unlocked CMP 170HX (SM80, ~63.4 GiB usable each) with TEP2 tensor+expert parallel and fixed MTP6 speculative decoding with per-request draft isolation. v0.3.0 supports 1-10 active requests (max_num_seqs accepts 1-10, defaults to 1 when omitted; repo example sets 10) with compact/topk vocabulary transport and expanded PCIe communication; history-copy (ngram hybrid) drafting removed. The ~51.2 GB FP8 PLE table stays on NVMe, read GPU-direct via GDS/cuFile with 32 persistent readers and 3 staging slots - no CPU-memory relay; optional host-read PLE backend off by default. FlashInfer PCIe IPC AllReduce, 36-token prefix matching, bayley/cmpunlocker BAR1/P2P driver + host GDS. Measured upstream (2026-10-11, thinking off): 10 requests x 5,000 tokens, 749.67 tok/s aggregate, 76.29/request; historical 2026-09-08 window 151.05-168.29 tok/s decode at 8K-128K inputs xhigh thinking.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":158.19999999999998863131622783839702606201171875,"prefill_tok_s":2059.90000000000009094947017729282379150390625,"updated_at":"2026-10-29 21:32:27.639988-06","commit_sha":"7d911c7dd2ce6b0a45d3c14791ee74036a43e7ae","content_hash":"247ee93d0bf55ac542edef7091844ac3df34a323","votes":0,"credit":[{"name":"nguyenthimy2022kg-alt","url":"https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6.json","alternate":"/r/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX TP2 DFlash2 K7 hybrid drafting 393K (alexvnesta)","summary":"Tensor-parallel successor to Flun's 2x CMP 170HX layer-split recipe: ExLlamaV3 TP2 over both 64 GB cards with a 655,360-token Q8 KV pool (392,960-token request cap), DFlash2 K7 speculative decoding plus a NEW hybrid drafting mode (H=15 draft tokens, temperature-0 outputs byte-identical on/off), two concurrent sessions with fair scheduling, idle sessions parked in system RAM (resume 1-3 s), a cross-session 26.8K-prefix cache (repeat TTFT 20.8 s -> 0.7 s), and image input (base64 data URLs, bwrap-sandboxed decode). TP backend NCCL 2.30.7 over GPU P2P after content-verified p2p_check; beats the native host-RAM backend by +4.35/+6.48 % decode and +10 % prefill (serial n=10); TP_BACKEND=native is the fallback. Author A/B on one machine (2026-10-12): 123.6 / 44.3 tok/s easy/hard text vs Flun layer-split 96.2 / 34.2 on the SAME box (+28 %), hybrid drafting 134-147 tok/s on code-edit/JSON. OpenAI-compatible 127.0.0.1:8012/v1, no auth (loopback/proxy). STOCK cards only: PCIe Gen2 x16, 180 W.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":123.599999999999994315658113919198513031005859375,"prefill_tok_s":1291.5,"updated_at":"2026-10-29 21:32:25.153466-06","commit_sha":"4a4d9f8d79ff001c6c845746708a86e4a3025350","content_hash":"32bd428f49f6a62760cd5c983a7324bfd38163eb","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX \u2014 DFlash2 K7 Q8 KV 384K (Flun)","summary":"First ExLlamaV3 serving recipe on 2x CMP 170HX: full-VRAM HBM-resident GLM-5.3-Flash EXL3 3.05bpw (~116.6 GiB across 2x64GB, no CPU/SSD tier) with DFlash2 K7 speculative decoding \u2014 drafter re-quantized locally BF16->EXL3 6bpw (~0.96 GiB) with an 8K-token GPU ring KV cache so a 384K request needs no 384K draft cache (bit-exact over ring wraps). Q8 quantized MLA latent KV, k_hcfuse fused decode kernels (SM80 JIT, shadow-check bit-exact), process-local monkeypatches only (engine tree unmodified), opt-in previous-request DFlash prefix reuse (8K repeat TTFT 7.9s->1.1s; general follow-up defect documented, not fixed), OpenAI-compatible API with native PP/TG + acceptance timings. Alternative stock MTP d2 profile (fp16 192K). Author-reported ~90 tok/s sustained, 110-120 at high acceptance (Reddit 1x0b1ws; no run file). Cards cap-modded PCIe x16 (Gen2 x8 vs stock x4), no P2P; single-request; Q8 changes numerics.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":90,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:24.781602-06","commit_sha":"f363025763f08eecd079592032038eb42acf00ae","content_hash":"f3d955e45e537ed75550188c286a6fed02089829","votes":1,"credit":[{"name":"Flun (recipe, stack, profiles, drafter conversion)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused decode kernels)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"MiaAI-Lab (GLM-5.3-Flash-EXL3-2x-DGX-Sparks \u2014 conversion reference)","url":"https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k"}},{"slug":"qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple","title":"Qwen3.8-Flash-Next W4A16 2x CMP 170HX TP2 DFlash2 host-RAM PLE","summary":"Qwen3.8-Flash-Next (125B MoE) INT4-expert W4A16 on two CMP 170HX cards, full 262,144-token context: 241/350/198 tok/s one user (structured/code/prose), 495 tok/s editing code, 895/987/770 across eight, up to 128 requests at once totaling 1,937 tok/s (DFlash 2; 1,966 MTP), 4,696 tok/s cold prefill, 1,475,836-token KV pool. Serves halt95's W4A16-Merlin checkpoint on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) with the 51 GB FP8 n-gram table in HOST RAM (cpu_offload; PLE_SHARED_DIR shares one copy across servers), the whole decode step one CUDA graph - built for a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3/2 schedule, 44% faster than MTP on code c1, 83% editing code) via engine-file patches plus RecoverSSM one-state GDN; optional KV offload restores an evicted 100k-token context in 0.89 s (21 s without). Measured 2026-10-18/23 from a fresh clone under Morrowmake's protocol (temp 0, 400-token pin, median-of-5 c1 / median-of-3 aggregates).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":253.19999999999998863131622783839702606201171875,"prefill_tok_s":0,"updated_at":"2026-10-29 20:26:39.946771-06","commit_sha":"d558ee307e9364af70f6a3c8317aa1db958be441","content_hash":"da435f81c05c69636771a5f32003ae1c413e0bf2","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-w4a16-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"halt95 (W4A16-Merlin checkpoint + companion 3090 engine; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound INT4 experts) + RadixArk (FP8 n-gram table) + DominikBucko (MTP-head packing)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/MiaAI-Lab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple.json","alternate":"/r/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple"}},{"slug":"qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode","title":"Qwen3.8-27B NVFP4A4-AWQ 1x RTX PRO 6000 SGLang Fast Decode","summary":"Recipe by AppNetDeveloper. ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ on 1x RTX PRO 6000 Blackwell (SM120) with stock SGLang 0.5.22 plus engine overlays, no fork: FR-Spec pruned draft lm_head for a quantized FP8 head (slices weight AND per-row scale along vocab, bit-identical kept rows); KDA NVFP4 SM120 GEMM allowlist extension (KN=5120x34816,17408x5120) 85-89% DRAM bandwidth at M<=16 vs 50-70% default; tuned triton_scaled_mm fp8_w8a8_channelwise JSONs. On 0.5.22 the overlays land as ONE patch, patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (patch -p1). NEXTN/MTP, FP8_e4m3 KV cache, HiCache, 40k hot-token draft list; hybrid GDN+MTP needs --enable-linear-replayssm-spec. Author bench 2026-10-14 (400 output tokens, unique prefixes): 131.7 tok/s 1req 45k ctx vs 98.5 (+34%); 4req@45k +13.5%; 8req@27k +3%; 1req@141k +24%; same quality, no extra memory. MTP5 + FR-Spec is the author production default; measured on real agent text: 149 vs 140 @1req, 117 vs 103 @2req, 77 vs 69 @4req over MTP4.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":131.69999999999998863131622783839702606201171875,"prefill_tok_s":0,"updated_at":"2026-10-28 16:25:30.123672-06","commit_sha":"607c8d8babf1cd4237a56128e0614aceb2176332","content_hash":"ba21205503ad8f52405ab4becff180d4e348b45f","votes":0,"credit":[{"name":"AppNetDeveloper","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode"},{"name":"SGLang 0.5.22 port + MTP5/FR-Spec real-text results @ c00cfec525 (published upstream 2026-10-21)","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode/commit/c00cfec525efb20e9c4185ca6c86e61a3235dbc1"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode.json","alternate":"/r/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode"}},{"slug":"qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes","title":"Qwen3.8-Flash-Next W4A16 2x CMP 170HX PP2 MTP5 PLE (AlexBrandes)","summary":"Recipe by AlexBrandes. Qwen3.8-Flash-Next (Intel/Qwen3.8-Flash-Next-W4A16-AutoRound @ 4c67bf68) on two NVIDIA CMP 170HX cards in PIPELINE parallel (PP2, layer split 26/22) with a vLLM v0.30.0 fork (AlexBrandes/vllm @ 02acf67c, 6 commits) in a digest-pinned container. Native MTP with 5 draft tokens (lm_head quantized to int4 by tools/quantize_lm_head.py, sha256 pinned), reasoning + tool calls, full 262,144-token window. The ~102 GB n-gram (PLE) table stays on NVMe read through mmap (VLLM_PLE_MMAP=1); BF16 KV, prefix caching, FP32 Mamba state. FIRST 2x170HX recipe with NO P2P or link modification needed - PP sends one activation per step, so it runs on stock PCIe Gen2 x4 links (all sibling rows need the unlock/P2P-mod prerequisite). Measured 2026-10-17 with sparkDash (3e6d4ef): 226 tok/s single-user code, 840 tok/s 8 users, 5,885-7,104 tok/s prefill 4k-128k; HumanEval 157/164, GSM8K 1,271/1,319, needle 12/12.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":226.19999999999998863131622783839702606201171875,"prefill_tok_s":0,"updated_at":"2026-10-27 20:30:16.824026-06","commit_sha":"44b6d37a739c1ee1ea463ffb4775b87382ef4c39","content_hash":"035586de62fecf1d4dec99de529bd4f53e8a3aa5","votes":0,"credit":[{"name":"AlexBrandes (recipe, fork, image, measurements)","url":"https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next"},{"name":"AlexBrandes (vllm fork: PP + NVMe-mmapped PLE, Mamba PP state, draft lm_head, graph-safe PLE, bounded prefill)","url":"https://github.com/AlexBrandes/vllm/tree/cmp170hx-qwen38-flash-next"},{"name":"polyuij42-del (community CMP 170HX W4A16 work the fork builds on)","url":"https://github.com/polyuij42-del/qwen38-flash-next-w4a16-cmp170hx"},{"name":"nguyenthimy2022kg-alt (community SM80 170HX work the fork builds on)","url":"https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX"},{"name":"MiaAI-Lab (sparkDash benchmark protocol @ 3e6d4ef)","url":"https://github.com/MiaAI-Lab/sparkDash"},{"name":"Intel (INT4 W4A16 AutoRound quant)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes.json","alternate":"/r/qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes"}},{"slug":"qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang","title":"Qwen3.8-27B Bit-Exact Spec-Decode 1x RTX PRO 6000 SGLang","summary":"Recipe by bill-h-lin. Qwen3.8-27B BF16 (BF16 KV, temperature 0) on one RTX PRO 6000 Blackwell Max-Q (SM120) with SGLang made bit-exact under speculative decoding: verify GDN recurrence routed through the decode kernel (old verify kernel tiled state differently, kept beta fp32), verify attention run as decode rows, a DeepGEMM sm120 bf16 small-M GEMM fence for repeatability (upstream from 0.2.3), and upstream sgl-project/sglang#42145 routing small-M bf16 GEMMs to a bit-identical Triton tile to escape the DeepGEMM sm120 bandwidth cliff. Base: SGLang main 542a043 (2026-09-23), DeepGEMM 0.2.0. DFlash drafter (incoai/Qwen3.8-27B-DFlash2) at 8 draft tokens: 106.6 tok/s exact vs 109.8 non-exact vLLM short, 73.9 vs 67.4 at 2.2k, 42.6 vs 56.1 at 128k. Receipts: 24/24 token-id+logprob parity vs no-speculation (forced-reject, 8 in-flight, 128k); 0/1200 run-to-run divergences with the fence; divergences cluster where top-1/top-2 logits are within 0.25 (0.125 for backend/--max-num-seqs).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":73.900000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-27 20:30:16.519777-06","commit_sha":"d47db89f9ef31a0a96530e29d246e7f562509ee6","content_hash":"5fd4ef10f7e92f70e8b5831e5c33c58cfa0d635c","votes":0,"credit":[{"name":"bill-h-lin","url":"https://github.com/bill-h-lin/sglang-exact-spec-decode"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang.json","alternate":"/r/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang"}}]}