{"meta":{"count":28,"page":1,"pages":3,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["170hx-x2","170hx-x4","rtx-3090"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T15:25:56+00:00"},"recipes":[{"slug":"deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake","title":"DeepSeek-V4.1-Flash-1017 Hybrid EXL3 TP4 4x CMP 170HX Morrowmake","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.0 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw export, every other weight from DeepSeek's official FP8 1017 checkpoint (sha256-validated link build, no copies), and the model's ~189 GiB Engram tables in host RAM (VLLM_ENGRAM_PLACEMENT=auto, SSD fallback below ~212 GiB available). up to 319 tok/s one user structured (287 code / 162 prose), up to 765 across eight, up to 3,818 tok/s cold 2K prefill and a 1,247,406-token KV pool at 262,144 context, full-precision BF16 KV. Engine: Morrowmake's vLLM fork @ a0c57b52 on upstream 4e0681ed56 (sm_80 EXL3 + Marlin kernels, custom P2P all-reduce after a startup content check, DSpark speculative decoding 7/5 adaptive), pinned container image, TP4 with whole routed experts per card, optimised for PCIe x16 links. HumanEval 157/164, GSM8K 1,280/1,319, within test noise of the unmodified-checkpoint reference.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":162,"prefill_tok_s":3804,"updated_at":"2026-10-30 08:13:13.515043-06","commit_sha":"1d43d37b0b9cffa384845e62bcd3af425c04088c","content_hash":"79c2a1c9569e13d38af257c3a7c84e011e8a9cbe","votes":0,"credit":[{"name":"Morrowmake (recipe, engine fork, measurements)","url":"https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe"},{"name":"LibertAI - EXL3 2.9-bpw routed-expert quantization","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"cmpunlocker (amoghmunikote) - P2P driver basis + minimal TRAP31 patch","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"MiaAI-Lab - decode benchmark protocol (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"Marlin (IST-DASLab + Neural Magic contributors) - kernel basis","url":"https://github.com/IST-DASLab/marlin"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake.json","alternate":"/r/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake"}},{"slug":"glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash max effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records max reasoning effort, selected per request with model GLM-5.3-Flash-max. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":41.5,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:37.835428-06","commit_sha":"663174ab27ccbc0f93296953b897b19e08b64726","content_hash":"1bf0157fd231d61ac68175f7a1493df8e9583746","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash low effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records low reasoning effort, selected per request with model GLM-5.3-Flash-low. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":41.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:37.671135-06","commit_sha":"7a55561ae92fcdeb4a011950b04c4848b7666861","content_hash":"1c7b1a323fae222fd633e500b576c9f1fe1e7cec","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141","title":"GLM-5.3-Flash on 4x CMP 170HX \u2014 Morrowmake recipe 1.8.0, TP4 DFlash2","summary":"PP4 (pipeline-parallel 4, TP1) as-run on a 4x CMP 170HX rig 2026-09-28 at --gpu-memory-utilization 0.93 after 0.95 OOM-killed warmup (a 72 MiB KDA prefill workspace did not fit in the ~70 MiB left on the last card). Author 1.8.0 measurements below are TP4 unless labeled PP4; KV pool 2,088,960 tokens under PP4 (author-published 1.8.0 figure). Recipe release 1.8.0 2026-10-16 (engine 819146023f, image engine-819146023f) \u2014 updated from 1.7.2; engine rebased on upstream vLLM 4e0681ed56, same context/batch/fair-prefill defaults; 21 engine-switch env names renamed (old names silently ignored at code defaults); container-only install; author re-measured throughput/KV tables at 1.8.0.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":264.3999999999999772626324556767940521240234375,"prefill_tok_s":6264,"updated_at":"2026-10-29 23:29:37.313564-06","commit_sha":"6e47f035b02d04cee06466adb7fb69b21d9d5240","content_hash":"171456f9f739f5c140a9a08da526b95e056ea9fe","votes":1,"credit":[{"name":"Morrowmake","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141.json","alternate":"/r/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141"}},{"slug":"glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash high effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records high reasoning effort, selected per request with model GLM-5.3-Flash-high. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":43.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:26.881493-06","commit_sha":"26ce83d34683309b83e4478edb3ed1213018d87f","content_hash":"58f06fe95f6c37780466d207181753f0f7c5b12f","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81","title":"DeepSeek-V4.1-Flash-1017 Fidelity-First 4x CMP 170HX alanoo81","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on 4x CMP 170HX (sm_80, 64 GB each, PCIe Gen2 x16), TP4, as a fidelity-first variant of the GNmac-lab recipe 1.2.0 @ 2ad5741ab1 with four changes: routed experts of layers 18-22 grafted 2-bit to 3-bit (LibertAIDAI EXL3 3.5bpw shards 00020-00024-of-00044 into the 2.9bpw pack, all 40 layers 3-bit); no DSpark drafter (SPEC_TOKENS=0, frees ~2.8 GiB/card for KV); Engram tables genuinely loaded (ENGRAM_STORAGE=ssd from a 200 GiB RAM tmpfs); gpu_memory_utilization 0.98, KV pool 2,717,420 tokens at 262,144 context. Author-measured: 300/300 exact identifier copies, byte-identical repeats; perplexity -1.80% overall / -4.45% French vs the 2.9bpw pack (95% CIs exclude zero); ~105 tok/s decode c1, 449-560 tok/s aggregate c8; cold prefill 2,514 tok/s median. Documents a silent failure: leaving DSV41_ENGRAM_STORAGE unset (host mode) leaves the Engram tables unfilled - layers 1 and 14 act as identity functions, 15/20 copies, temp-0 output drift, no error.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":105,"prefill_tok_s":2514,"updated_at":"2026-10-29 21:57:43.459028-06","commit_sha":"5af1460f4a2dce4f590f6fdfcd108119285e3e7c","content_hash":"67eb1810e7eea071e85e8edab747fd98247dd872","votes":0,"credit":[{"name":"alanoo81 (recipe, graft, measurements)","url":"https://github.com/alanoo81/deepseek-v41-flash-4x-cmp170hx"},{"name":"GNmac-lab (base recipe 1.2.0 @ 2ad5741ab1, MIT)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"LibertAIDAI (EXL3 2.9bpw @ 0be63c33 / 3.5bpw @ 04a492fc)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"Curtis White (hybrid layout + quantization_config.json)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"vcruz305/vllm-exl3 (AGPL-3.0-only, inside the built image)","url":"https://github.com/vcruz305/vllm-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81.json","alternate":"/r/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81"}},{"slug":"qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3","title":"Qwen3.8-Flash-Next W4A16 INT4PLE on 2x modded CMP 170HX \u2014 PP2 MTP3 (DUFRelic)","summary":"Production 2x CMP 170HX recipe for Qwen3.8-Flash-Next: zankich INT4PLE repackage of halt95 Merlin W4A16 (94.4 GiB, 51B-param PLE table repacked INT4) on stock vLLM 0.30.0 + a 6-layer hash-guarded overlay stack (PP2/PLE #56444, KV filter #54793, MTP-over-PP #54709, slot statefix #55506, SpeedSuite 5-PR overlay, batched-P2P). PP2/TP1, MTP 3-token draft (measured 52.27% accept). Hardware is the recipe's own mod stack: 74-SM unlock (cmpunlocker), VBIOS 92.00.6D.00.0A, 170tune 1420 MHz SM cap + HBM 1728 MHz undervolt, BAR1 P2P driver patches, 140 W/card. Author-measured 2026-10-03 (median of 3, isolation audit): prefill C1 6213.7 tok/s @16k, decode C1 133.4 tok/s (SD 22.8 reported), decode C4 agg 245.2 tok/s; 55/56 C at 160 W peaks, no Xid/OOM. INT4 PLE table (~26.8 GiB) pinned in HOST RAM; KV BF16 (pinned build's QSA path rejects FP8). Driver mods optional for function but required for the published throughput.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":133.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:28.370851-06","commit_sha":"c26a7fd62ded418a43f5a10abf82eaedb7afc6bc","content_hash":"7c6051a80e2a07b8bca384eb300935641c88349b","votes":0,"credit":[{"name":"DUFRelic","url":"https://github.com/DUFRelic/qwen38-flash-next-recipe"},{"name":"zankich (INT4-PLE repackage + plugin)","url":"https://github.com/zankich/vllm"},{"name":"halt95 (W4A16-Merlin base checkpoint)","url":"https://github.com/halt95/merlin-engine"},{"name":"michael-lgtm (public overlay rebuild)","url":"https://github.com/michael-lgtm/qwen38-flash-next-cmp-overlay"},{"name":"amoghmunikote (cmpunlocker SM unlock + BAR1 P2P)","url":"https://github.com/amoghmunikote/cmpunlocker"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3.json","alternate":"/r/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3"}},{"slug":"qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6","title":"Qwen3.8-Flash-Next NVFP4 2x CMP 170HX TEP2 GDS SSD PLE MTP6","summary":"Recipe by nguyenthimy2022kg-alt. Community patched vLLM run of the NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on 2x unlocked CMP 170HX (SM80, ~63.4 GiB usable each) with TEP2 tensor+expert parallel and fixed MTP6 speculative decoding with per-request draft isolation. v0.3.0 supports 1-10 active requests (max_num_seqs accepts 1-10, defaults to 1 when omitted; repo example sets 10) with compact/topk vocabulary transport and expanded PCIe communication; history-copy (ngram hybrid) drafting removed. The ~51.2 GB FP8 PLE table stays on NVMe, read GPU-direct via GDS/cuFile with 32 persistent readers and 3 staging slots - no CPU-memory relay; optional host-read PLE backend off by default. FlashInfer PCIe IPC AllReduce, 36-token prefix matching, bayley/cmpunlocker BAR1/P2P driver + host GDS. Measured upstream (2026-10-11, thinking off): 10 requests x 5,000 tokens, 749.67 tok/s aggregate, 76.29/request; historical 2026-09-08 window 151.05-168.29 tok/s decode at 8K-128K inputs xhigh thinking.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":158.19999999999998863131622783839702606201171875,"prefill_tok_s":2059.90000000000009094947017729282379150390625,"updated_at":"2026-10-29 21:32:27.639988-06","commit_sha":"7d911c7dd2ce6b0a45d3c14791ee74036a43e7ae","content_hash":"247ee93d0bf55ac542edef7091844ac3df34a323","votes":0,"credit":[{"name":"nguyenthimy2022kg-alt","url":"https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6.json","alternate":"/r/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX TP2 DFlash2 K7 hybrid drafting 393K (alexvnesta)","summary":"Tensor-parallel successor to Flun's 2x CMP 170HX layer-split recipe: ExLlamaV3 TP2 over both 64 GB cards with a 655,360-token Q8 KV pool (392,960-token request cap), DFlash2 K7 speculative decoding plus a NEW hybrid drafting mode (H=15 draft tokens, temperature-0 outputs byte-identical on/off), two concurrent sessions with fair scheduling, idle sessions parked in system RAM (resume 1-3 s), a cross-session 26.8K-prefix cache (repeat TTFT 20.8 s -> 0.7 s), and image input (base64 data URLs, bwrap-sandboxed decode). TP backend NCCL 2.30.7 over GPU P2P after content-verified p2p_check; beats the native host-RAM backend by +4.35/+6.48 % decode and +10 % prefill (serial n=10); TP_BACKEND=native is the fallback. Author A/B on one machine (2026-10-12): 123.6 / 44.3 tok/s easy/hard text vs Flun layer-split 96.2 / 34.2 on the SAME box (+28 %), hybrid drafting 134-147 tok/s on code-edit/JSON. OpenAI-compatible 127.0.0.1:8012/v1, no auth (loopback/proxy). STOCK cards only: PCIe Gen2 x16, 180 W.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":123.599999999999994315658113919198513031005859375,"prefill_tok_s":1291.5,"updated_at":"2026-10-29 21:32:25.153466-06","commit_sha":"4a4d9f8d79ff001c6c845746708a86e4a3025350","content_hash":"32bd428f49f6a62760cd5c983a7324bfd38163eb","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX \u2014 DFlash2 K7 Q8 KV 384K (Flun)","summary":"First ExLlamaV3 serving recipe on 2x CMP 170HX: full-VRAM HBM-resident GLM-5.3-Flash EXL3 3.05bpw (~116.6 GiB across 2x64GB, no CPU/SSD tier) with DFlash2 K7 speculative decoding \u2014 drafter re-quantized locally BF16->EXL3 6bpw (~0.96 GiB) with an 8K-token GPU ring KV cache so a 384K request needs no 384K draft cache (bit-exact over ring wraps). Q8 quantized MLA latent KV, k_hcfuse fused decode kernels (SM80 JIT, shadow-check bit-exact), process-local monkeypatches only (engine tree unmodified), opt-in previous-request DFlash prefix reuse (8K repeat TTFT 7.9s->1.1s; general follow-up defect documented, not fixed), OpenAI-compatible API with native PP/TG + acceptance timings. Alternative stock MTP d2 profile (fp16 192K). Author-reported ~90 tok/s sustained, 110-120 at high acceptance (Reddit 1x0b1ws; no run file). Cards cap-modded PCIe x16 (Gen2 x8 vs stock x4), no P2P; single-request; Q8 changes numerics.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":90,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:24.781602-06","commit_sha":"f363025763f08eecd079592032038eb42acf00ae","content_hash":"f3d955e45e537ed75550188c286a6fed02089829","votes":1,"credit":[{"name":"Flun (recipe, stack, profiles, drafter conversion)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused decode kernels)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"MiaAI-Lab (GLM-5.3-Flash-EXL3-2x-DGX-Sparks \u2014 conversion reference)","url":"https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin","title":"GLM-5.3-Flash EXL3 3.05bpw on 1x RTX 3090 \u2014 RAM/NVMe expert tiering (near-exact)","summary":"320B MoE served on ONE 24 GB RTX 3090 with stock exllamav3 1.5.1 plus process-local monkeypatches: the 12,096 routed experts (42 MoE layers x 288, 8 picked per token per layer) tier across VRAM, pinned host RAM and a packed NVMe store. An elastic VRAM cache holds the ~1,300 hottest experts (12.4 GB); evicted experts move into a pinned no-duplicates RAM tier sized from the container cap; a 117 GB O_DIRECT record store covers the rest, with layer-ahead prefetch, a victim write-back ring and an AVX2 CPU lane computing RAM-resident decode picks in place. Modes: fast (all experts in RAM, ~218 GiB free, 28.2 tok/s, KL 0.005), nvme (55 GiB cap, 19.5 tok/s, KL 0.008; 16 GiB cap 14.7 lab / 14.38 sweep), nvme-exact (bit-exact, 8.3 tok/s), and an experimental Intel Arc Pro B70 peer-offload tier (GLM53_B70=1, 26.5 tok/s, KL 0.003 - not lab-run). 131,072-token KV, OpenAI API on port 30000, model id glm-5.3-flash, no output caps. Registry lab six gates pass at 55 GiB (PR #199); 16 GiB reported.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":24.10000000000000142108547152020037174224853515625,"prefill_tok_s":1015,"updated_at":"2026-10-29 21:32:24.549403-06","commit_sha":"53520bbf5e06f65db6670e8693efb018f75026c8","content_hash":"65ab07a0787590e7c21331b006d4a13553080d02","votes":0,"credit":[{"name":"sybil-solutions (image, tiering runtime, omarchy plugin)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"turboderp (ExLlamaV3 + EXL3 3.05bpw checkpoint)","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin"}},{"slug":"deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash EXL3 3bpw TC-MoE on 4x CMP 170HX TP4 (original checkpoint)","summary":"DeepSeek-V4.1-Flash (ORIGINAL deepseek-ai/DeepSeek-V4.1-Flash checkpoint, quantized by Mia-AiLab as EXL3 3.02bpw \u2014 NOT the 2026-10-17 '1017' re-release, which has its own sibling recipe) served EXL3 3bpw tensor-parallel 4 on four CMP 170HX cards (sm_80, 140 W, HBM locked 1728 MHz) in a 2x Xeon E5-2689 v4 Broadwell host, PCIe Gen2 x16, no NVLink, using Morrowmake's tensor-core MoE kernels + host-staged all-reduce ported onto this checkpoint: K3 on every served layer with a split K2 down-projection, row-rule hybrid (coop rows 1-42, moe_out_v2 43-64, moe_out_prefill 65-2048), one shared 540.4 MiB MoE workspace per GPU, zero-register-spill served kernels. Measured by the author: 178.3 tok/s structured decode c1, 587 tok/s across 8 users, median TTFT 8.37 s from 2K through 255K context (10.9 s at 255,424) against the same host's optimized graphs+coop+NCCL stack at 172.7 / 456 / 10.15 s; needles 9/9 through 256K; KV pool 616,916 tokens.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":149.19999999999998863131622783839702606201171875,"prefill_tok_s":1486,"updated_at":"2026-10-29 21:32:23.373585-06","commit_sha":"dc2dbe7cce66fb0f4d3dfad6b6765ed3c4ea8f74","content_hash":"5640bd33a3ca7d6a71e5a4804e0ec780e0c34fb1","votes":0,"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4"}}]}