{"meta":{"count":17,"page":1,"pages":2,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["170hx","170hx-x4","rtx-3090-x2"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T14:33:18+00:00"},"recipes":[{"slug":"deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake","title":"DeepSeek-V4.1-Flash-1017 Hybrid EXL3 TP4 4x CMP 170HX Morrowmake","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.0 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw export, every other weight from DeepSeek's official FP8 1017 checkpoint (sha256-validated link build, no copies), and the model's ~189 GiB Engram tables in host RAM (VLLM_ENGRAM_PLACEMENT=auto, SSD fallback below ~212 GiB available). up to 319 tok/s one user structured (287 code / 162 prose), up to 765 across eight, up to 3,818 tok/s cold 2K prefill and a 1,247,406-token KV pool at 262,144 context, full-precision BF16 KV. Engine: Morrowmake's vLLM fork @ a0c57b52 on upstream 4e0681ed56 (sm_80 EXL3 + Marlin kernels, custom P2P all-reduce after a startup content check, DSpark speculative decoding 7/5 adaptive), pinned container image, TP4 with whole routed experts per card, optimised for PCIe x16 links. HumanEval 157/164, GSM8K 1,280/1,319, within test noise of the unmodified-checkpoint reference.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":162,"prefill_tok_s":3804,"updated_at":"2026-10-30 08:13:13.515043-06","commit_sha":"1d43d37b0b9cffa384845e62bcd3af425c04088c","content_hash":"79c2a1c9569e13d38af257c3a7c84e011e8a9cbe","votes":0,"credit":[{"name":"Morrowmake (recipe, engine fork, measurements)","url":"https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe"},{"name":"LibertAI - EXL3 2.9-bpw routed-expert quantization","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"cmpunlocker (amoghmunikote) - P2P driver basis + minimal TRAP31 patch","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"MiaAI-Lab - decode benchmark protocol (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"Marlin (IST-DASLab + Neural Magic contributors) - kernel basis","url":"https://github.com/IST-DASLab/marlin"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake.json","alternate":"/r/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake"}},{"slug":"glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141","title":"GLM-5.3-Flash on 4x CMP 170HX \u2014 Morrowmake recipe 1.8.0, TP4 DFlash2","summary":"PP4 (pipeline-parallel 4, TP1) as-run on a 4x CMP 170HX rig 2026-09-28 at --gpu-memory-utilization 0.93 after 0.95 OOM-killed warmup (a 72 MiB KDA prefill workspace did not fit in the ~70 MiB left on the last card). Author 1.8.0 measurements below are TP4 unless labeled PP4; KV pool 2,088,960 tokens under PP4 (author-published 1.8.0 figure). Recipe release 1.8.0 2026-10-16 (engine 819146023f, image engine-819146023f) \u2014 updated from 1.7.2; engine rebased on upstream vLLM 4e0681ed56, same context/batch/fair-prefill defaults; 21 engine-switch env names renamed (old names silently ignored at code defaults); container-only install; author re-measured throughput/KV tables at 1.8.0.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":264.3999999999999772626324556767940521240234375,"prefill_tok_s":6264,"updated_at":"2026-10-29 23:29:37.313564-06","commit_sha":"6e47f035b02d04cee06466adb7fb69b21d9d5240","content_hash":"171456f9f739f5c140a9a08da526b95e056ea9fe","votes":1,"credit":[{"name":"Morrowmake","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141.json","alternate":"/r/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141"}},{"slug":"deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81","title":"DeepSeek-V4.1-Flash-1017 Fidelity-First 4x CMP 170HX alanoo81","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on 4x CMP 170HX (sm_80, 64 GB each, PCIe Gen2 x16), TP4, as a fidelity-first variant of the GNmac-lab recipe 1.2.0 @ 2ad5741ab1 with four changes: routed experts of layers 18-22 grafted 2-bit to 3-bit (LibertAIDAI EXL3 3.5bpw shards 00020-00024-of-00044 into the 2.9bpw pack, all 40 layers 3-bit); no DSpark drafter (SPEC_TOKENS=0, frees ~2.8 GiB/card for KV); Engram tables genuinely loaded (ENGRAM_STORAGE=ssd from a 200 GiB RAM tmpfs); gpu_memory_utilization 0.98, KV pool 2,717,420 tokens at 262,144 context. Author-measured: 300/300 exact identifier copies, byte-identical repeats; perplexity -1.80% overall / -4.45% French vs the 2.9bpw pack (95% CIs exclude zero); ~105 tok/s decode c1, 449-560 tok/s aggregate c8; cold prefill 2,514 tok/s median. Documents a silent failure: leaving DSV41_ENGRAM_STORAGE unset (host mode) leaves the Engram tables unfilled - layers 1 and 14 act as identity functions, 15/20 copies, temp-0 output drift, no error.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":105,"prefill_tok_s":2514,"updated_at":"2026-10-29 21:57:43.459028-06","commit_sha":"5af1460f4a2dce4f590f6fdfcd108119285e3e7c","content_hash":"67eb1810e7eea071e85e8edab747fd98247dd872","votes":0,"credit":[{"name":"alanoo81 (recipe, graft, measurements)","url":"https://github.com/alanoo81/deepseek-v41-flash-4x-cmp170hx"},{"name":"GNmac-lab (base recipe 1.2.0 @ 2ad5741ab1, MIT)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"LibertAIDAI (EXL3 2.9bpw @ 0be63c33 / 3.5bpw @ 04a492fc)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"Curtis White (hybrid layout + quantization_config.json)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"vcruz305/vllm-exl3 (AGPL-3.0-only, inside the built image)","url":"https://github.com/vcruz305/vllm-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81.json","alternate":"/r/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81"}},{"slug":"qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm","title":"Qwen3.8-Flash-Next 3bpw FP8-PLE 1x CMP 170HX DFlash2 RecoverSSM","summary":"Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s editing code, 799.5/791.3/586.4 across eight, up to 48 requests at once - 1,384 tok/s total with the default DFlash2 drafter (1,403 with the checkpoint's MTP head), 4,243 tok/s cold prefill, 292,784-token KV pool. Serves klee100's 3 bpw AutoRound checkpoint (3-bit g128 / 2-bit g64 experts, 47 GiB on card) with halt95's FP8 n-gram table in HOST RAM (48 GiB, hard-linked at build) on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) so the whole decode step is one CUDA graph on a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3 by request count) + the recipe's RecoverSSM + BF16 GDN-state patches (9 files mounted over the image) fit 48 concurrent requests on one card. Measured 2026-10-23 under the MiaAI-Lab protocol.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":197.80000000000001136868377216160297393798828125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.955646-06","commit_sha":"af7d7dde12376551476c8b9e36cd72c68dc32408","content_hash":"c82e80ffa9955d905b61bda381cf7d89db1350ba","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/Mia-AiLab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json","alternate":"/r/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm"}},{"slug":"deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash EXL3 3bpw TC-MoE on 4x CMP 170HX TP4 (original checkpoint)","summary":"DeepSeek-V4.1-Flash (ORIGINAL deepseek-ai/DeepSeek-V4.1-Flash checkpoint, quantized by Mia-AiLab as EXL3 3.02bpw \u2014 NOT the 2026-10-17 '1017' re-release, which has its own sibling recipe) served EXL3 3bpw tensor-parallel 4 on four CMP 170HX cards (sm_80, 140 W, HBM locked 1728 MHz) in a 2x Xeon E5-2689 v4 Broadwell host, PCIe Gen2 x16, no NVLink, using Morrowmake's tensor-core MoE kernels + host-staged all-reduce ported onto this checkpoint: K3 on every served layer with a split K2 down-projection, row-rule hybrid (coop rows 1-42, moe_out_v2 43-64, moe_out_prefill 65-2048), one shared 540.4 MiB MoE workspace per GPU, zero-register-spill served kernels. Measured by the author: 178.3 tok/s structured decode c1, 587 tok/s across 8 users, median TTFT 8.37 s from 2K through 255K context (10.9 s at 255,424) against the same host's optimized graphs+coop+NCCL stack at 172.7 / 456 / 10.15 s; needles 9/9 through 256K; KV pool 616,916 tokens.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":149.19999999999998863131622783839702606201171875,"prefill_tok_s":1486,"updated_at":"2026-10-29 21:32:23.373585-06","commit_sha":"dc2dbe7cce66fb0f4d3dfad6b6765ed3c4ea8f74","content_hash":"5640bd33a3ca7d6a71e5a4804e0ec780e0c34fb1","votes":0,"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4"}},{"slug":"deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw TC-MoE on 4x CMP 170HX TP4","summary":"DeepSeek-V4.1-Flash-1017 (the 2026-10-17 re-release checkpoint, NOT the original DeepSeek-V4.1-Flash) as a hybrid quant on the TC-MoE port: official deepseek-ai/DeepSeek-V4.1-Flash-1017 FP8 tensors for attention, shared experts, head, DSpark draft and Engram (shards 47/48), with LibertAIDAI/...1017-EXL3-2.9bpw routed experts (layers 0-39, mostly 3-bit, layers 18-22 2-bit) TP4 on four CMP 170HX cards (sm_80, 140 W, ~62.5 GB/card). Same Morrowmake kernels + image as the EXL3 3bpw TC-MoE recipe; VLLM_EXL3_COOP=1; EXL3 arena cache (~1.1 GiB/layer, load shortcut only). 1017 quantization_config.json is the switch (dsv41_main_routed_experts_only, mtp_experts=source, non-routed stays FP8, compressor projection BF16). Author-measured: 180.4/137.1/96.6 tok/s structured/code/prose c1; 8-user aggregate 621.1/376.1/282.5 vs 564.7 previous Mia 3.0bpw. Cold 131k prompt 1,330 prefill / 163.9 decode. Needles 9/9. KV pool 1,534,065 tokens; vision on, drafter text-only.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":165.599999999999994315658113919198513031005859375,"prefill_tok_s":1390.09999999999990905052982270717620849609375,"updated_at":"2026-10-29 21:32:22.449013-06","commit_sha":"306225cea85cf158c3036c971b031a9c1740075b","content_hash":"59b7b30fa1a9d7fe2197055fbfc51c994b611ccc","votes":0,"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"Measured serve used local image sha256:55d2bd9a6664 (no published image)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"PixelML/sm80vllm branch sm80 (DeepseekV4AmpereMLAAttention) - SM80 attention for this model","url":"https://github.com/PixelML/sm80vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4"}},{"slug":"deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx","title":"DeepSeek-V4.1-Flash-1017 Engram-NVMe TP4/PP4 4x CMP 170HX","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards power-capped at 180 W, with the model's 189 GiB Engram n-gram tables streamed from NVMe on demand. Recipe 1.2.0 (author measurements, claimed_by=submitter T0): 355.7/258.4/183.6 tok/s one user (structured/code/prose), 867.6/544.0/440.5 across eight, 2,416 tok/s cold prefill (TP4; PP4 2,311), 1,395,731-token KV pool at 262,144 context (1,270,789 at 1M), HumanEval 154/164 (6 length-capped; McNemar p>=0.5 vs 156/164), GSM8K 1,275/1,319 (TP4). Hybrid checkpoint assembled without copying tensors: LibertAI EXL3 2.9-bpw routed experts (layers 0-39) + official FP8 attention/shared experts/head/DSpark drafter/Engram shards + BF16 embeddings/LM head/vision, fp8_ds_mla KV. Engine: vLLM on the community sm_80 stack (zebgop-ops sm80 sparse attention + DSpark V1 + Engram row store, FlashInfer PCIe IPC all-reduce) plus this recipe's measured keep-or-drop patches. TP4 (default, needs x16 + P2P) and PP4 (x1/x4, no P2P) from one .env switch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":150.599999999999994315658113919198513031005859375,"prefill_tok_s":0,"updated_at":"2026-10-28 23:26:24.539707-06","commit_sha":"1b9a60d70c9e1e1f83c7db45556ac3450d511600","content_hash":"dadcb54c6118ad769c07ea0930690a53cead0c65","votes":0,"credit":[{"name":"GNmac-lab / greencandle (recipe, patches, measurements)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"CurtisAccelerate - TP4 hybrid-checkpoint recipe + tensor-core EXL3 MoE port","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE"},{"name":"zebgop-ops - dsv41reap-pp sm_80 sparse attention, PP shadow KV, DSpark V1, Engram row store","url":"https://github.com/zebgop-ops/dsv41reap-pp"},{"name":"vcruz305 - vllm-exl3 (AGPL-3.0-only)","url":"https://github.com/vcruz305/vllm-exl3"},{"name":"LibertAI - EXL3 2.9-bpw quantization (MIT)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"kaka86mm - dsv41-flash-pp4-170hx (watched sibling)","url":"https://github.com/kaka86mm/dsv41-flash-pp4-170hx"},{"name":"Morrowmake - CMP 170HX recipe format, P2P work, tensor-core MoE kernels","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"Mia-AiLab - decode benchmark protocol + prompts (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"FlashInfer - PCIe IPC all-reduce","url":"https://github.com/flashinfer-ai/flashinfer"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx.json","alternate":"/r/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx"}},{"slug":"deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram","title":"DeepSeek-V4.1-Flash-1017 dsx 2.6-bit GPTQ 2x RTX 3090 (Engram)","summary":"DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) quantized to 187.6 GB single GGUF (2.6127 bpw routed-expert average: 5717 PTQ1_0 / 2545 Q2_0 / 7098 Q3_K per-expert GPTQ with allocator-spent bit budget; dense Q8_0 attention/shared expert/head, F16 embedding/compressor/indexer/Engram-wkv, F32 norms/router/mHC), served by dsx, a Rust+CUDA engine built for this checkpoint, on two RTX 3090s with 192 GB RAM. ~26 tok/s decode / ~170 tok/s prefill at --ctx 262144 with all 15,360 routed experts resident. Exact routing: no expert pruning, no router bias - the rank file orders placement only. Engram n-gram tables (~203 GB) are NOT quantized: read at runtime from the official safetensor shards by a prefetch thread. Hottest ~3,300 experts in VRAM across both GPUs (GPU0's set adapts to the conversation, DSX_ADAPT=8 swaps/token), remaining experts computed on 14 CPU threads from a hugepage RAM arena. Quality vs FP8/FP4 teacher (dsx eval, Lucebox 1017 set): KL 0.1972, top-1 82.59%, PPL 9.3626.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":26,"prefill_tok_s":170,"updated_at":"2026-10-28 15:52:50.29183-06","commit_sha":"942f2d4b4ca3b3ca6697019daf09d9615a47b0af","content_hash":"57641e58a944d825b6d88744c42b9451d8ee916a","votes":0,"credit":[{"name":"lxe (dsx engine + 2bit reference quant)","url":"https://github.com/lxe/dsx"},{"name":"icecubetr (2.6-bit GPTQ quant + measurements)","url":"https://huggingface.co/icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit"},{"name":"deepseek-ai (DeepSeek-V4.1-Flash-1017 checkpoint)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"Lucebox (1017 eval set)","url":"https://huggingface.co/Lucebox/DeepSeek-V4.1-Flash-1017-ROCMFP-GGUF"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram.json","alternate":"/r/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram"}},{"slug":"qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy","title":"Qwen3.8-Flash-Next-FP8 4x CMP 170HX \u2014 PP4 MTP-4 patched vLLM (g4bwy)","summary":"Recipe by g4bwy. Official Qwen/Qwen3.8-Flash-Next-FP8 on 4x unlocked CMP 170HX (GA100 sm_80, 74 SMs, 64 GB) with PP=4 (layer partition 13,12,12,11) and MTP-4 speculative decoding on pinned vLLM mainline 183625d4f9 (2026-10-13, precompiled wheel 0.32.0rc1.dev34+g183625d4f) plus a 7-patch series applied with git am (tree hash must equal 18e0ec8942f61f82fd3a13bd9d9c6b9ce67ad371; 2 patches adopted from upstream PRs #56444/#58094) that unblocks PP+MTP+PLE on sm_80. 1M context at static YaRN 4.0, PLE n-gram table in pinned host RAM (--engram-config cpu_offload), Mamba align prefix caching with retention 16160 + match-unit 16 (99.8-99.9% hits), NCCL P2P disabled (host-staged wins at PP=4), expandable_segments, 200 W per-card caps, uv venv, no docker. Optional W8A16 int8-dense lane (checkpoint/) +15-20% single-stream.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":183.19999999999998863131622783839702606201171875,"prefill_tok_s":10038,"updated_at":"2026-10-20 21:52:36.721138-06","commit_sha":"540d404493ab67413a71efa0f5e541d18d329313","content_hash":"ab27af4ddefebc602e00353e6d7b640e515f9c4d","votes":0,"credit":[{"name":"g4bwy","url":"https://github.com/g4bwy/qwen3.8-flash-next-cmp-170hx"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy.json","alternate":"/r/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy"}},{"slug":"qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4","title":"Qwen3.8-27B AWQ-INT4 + DSpark on RTX 3090s \u2014 SGLang TP1/TP2/TP4","summary":"Validated SGLang serving for the dense Qwen3.8-27B hybrid GDN vision-language model (48 Gated-DeltaNet + 16 gated-attention layers) on RTX 3090 24 GB: one digest-pinned container whose TP env selects tuned 1x/2x/4x presets. AWQ-INT4 W4A16 (Marlin) beats FP8 on Ampere; DSpark speculative decoding (RadixArk 1.36B draft) on 2x/4x with auto-fallback on TP1. Image input everywhere, video on 2x/4x. Author-measured: 136.8 tok/s decode at 8k on 4x (169.8 aggregate at C8), 112.9 on 2x, 43.2 on 1x; up to 128k context on 4x.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":127.099999999999994315658113919198513031005859375,"prefill_tok_s":1372.700000000000045474735088646411895751953125,"updated_at":"2026-10-20 21:52:36.534482-06","commit_sha":"26a1363966d9065a310e9ec7caf124206a25f3f2","content_hash":"3fa45739100d87065553d5fb598424a66d4742fb","votes":0,"credit":[{"name":"sybil-solutions / 0xSero (SGLang presets, image build, benchmarks)","url":"https://github.com/sybil-solutions/qwen-serve"},{"name":"cyankiwi (AWQ-INT4 W4A16 checkpoint)","url":"https://huggingface.co/cyankiwi/Qwen3.8-27B-AWQ-INT4"},{"name":"RadixArk (DSpark speculative drafter)","url":"https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4.json","alternate":"/r/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x RTX 3090 \u2014 vLLM + ExLlamaV3 CPU expert offload, 256K (DominikBucko)","summary":"GLM-5.3-Flash (turboderp EXL3 3.05bpw) tensor-parallel on 2x RTX 3090 24 GB + 128 GB system RAM: vendor vLLM image with the spele1100/vllm-glm53 GLM-5.3 overlay, ExLlamaV3 @ 499890c with EXL3 MoE decode kernels, and CPU expert-offload pools (UVA exact host registers, per-layer LRU expert cache with the hottest 20% pinned, glm53_host.py pool planner, checkpoint lock + k-pool index installer). Bounded 4,096-token thinking budget + preserved reasoning for agent loops, 262,144-token context, prefix caching, OpenAI-compatible API. Docker image + systemd example, reproducible from repro.lock.json pins; measured up to 1,155 tok/s prefill and 47-50 tok/s decode with needles-at-32K/131K/260K + coherence gates passing.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":49.39999999999999857891452847979962825775146484375,"prefill_tok_s":1130.5,"updated_at":"2026-10-20 21:52:36.396387-06","commit_sha":"c93c82b83e6832d41dd10a7037c607a3f4b53d1b","content_hash":"c807eb1b13116b97fedcab43cef2aad9dca165e7","votes":0,"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/glm53-flash-2x3090"},{"name":"turboderp-org (EXL3 quant + ExLlamaV3)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"spele1100 (GLM-5.3 vLLM overlay)","url":"https://github.com/spele1100/vllm-glm53"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp"}},{"slug":"qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix","title":"Qwen3.8-27B INT8 MTP3 prefix-cache on 1x CMP 170HX Hynix","summary":"Qwen3.8-27B at 65-67 tok/s decode (74 structured) with MTP k=3 speculative decoding AND prefix caching simultaneously on a single NVIDIA CMP 170HX 64GB (Hynix HBM2e, GA100/sm80), 262,144-token context on fp8_e4m3 KV, 24/7 production on pinned upstream vLLM nightly 0.26.1rc1.dev1024+g1fe3a1571 \u2014 the first build where prefix caching and MTP spec decode coexist (GDN mamba-align). Cached TTFT on a 7K shared prefix drops ~4.0 s -> 1.44 s (33.5% hit rate) while decode gains 7-9% over the 0.27.1 fork fallback \u2014 the 'MTP vs prefix cache, pick one' tradeoff is gone. Load-bearing trio --async-scheduling + --max-num-batched-tokens 8192 + --max-num-scheduled-tokens 8192 (default chunked-prefill batch = 0.0% hit rate while looking healthy); --prefix-match-unit 400 halves cached TTFT further (must divide the 1600-token GDN block); batch budget 32000 OOMs and hard-wedges the card. Samsung-40GB side recipe (int8 + DFlash2, 97.8 tok/s) with an sm80 int8-repack CPU-fallback patch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":68.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-20 15:34:22.585629-06","commit_sha":"ad3f55dab02cedfb2b187f7dcfc3f1647e0aaa0c","content_hash":"68f95048b88c132d1ddbffc8ceceaf5c38be5140","votes":0,"credit":[{"name":"ahnguyen17 (recipe, patches, probes, A/B benchmarks)","url":"https://github.com/ahnguyen17/cmp-170hx-vllm"},{"name":"lued (Qwen3.8-27B-INT8-W8A16-MTP checkpoint)","url":"https://huggingface.co/lued/Qwen3.8-27B-INT8-W8A16-MTP"},{"name":"vllm-project (vLLM upstream; #50021 GDN spec bounds, #45238 align-mode cache retention)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json","alternate":"/r/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix"}}]}