{"meta":{"count":21,"page":1,"pages":2,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["rtx-3090","170hx-x2","170hx"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T20:43:25+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple","title":"Qwen3.8-Flash-Next W4A16 2x CMP 170HX TP2 DFlash2 host-RAM PLE","summary":"Qwen3.8-Flash-Next (125B MoE) INT4-expert W4A16 on two CMP 170HX cards, full 262,144-token context: 241/350/198 tok/s one user (structured/code/prose), 495 tok/s editing code, 895/987/770 across eight, up to 128 requests at once totaling 1,937 tok/s (DFlash 2; 1,966 MTP), 4,696 tok/s cold prefill, 1,475,836-token KV pool. Serves halt95's W4A16-Merlin checkpoint on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) with the 51 GB FP8 n-gram table in HOST RAM (cpu_offload; PLE_SHARED_DIR shares one copy across servers), the whole decode step one CUDA graph - built for a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3/2 schedule, 44% faster than MTP on code c1, 83% editing code) via engine-file patches plus RecoverSSM one-state GDN; optional KV offload restores an evicted 100k-token context in 0.89 s (21 s without). Measured 2026-10-18/23 from a fresh clone under Morrowmake's protocol (temp 0, 400-token pin, median-of-5 c1 / median-of-3 aggregates).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":241.19999999999998863131622783839702606201171875,"prefill_tok_s":0,"updated_at":"2026-10-31 11:51:46.269984-06","commit_sha":"33d1f11aee9d87ff9b04744be9ec577d953fa7c1","content_hash":"a970a21b9dad86241fc126beaaba1946d67d7b0e","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-w4a16-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"halt95 (W4A16-Merlin checkpoint + companion 3090 engine; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound INT4 experts) + RadixArk (FP8 n-gram table) + DominikBucko (MTP-head packing)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/MiaAI-Lab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple.json","alternate":"/r/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple"}},{"slug":"qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3","title":"Qwen3.8-Flash-Next-Uncensored AWQ W4A16 on 2x CMP 170HX \u2014 PP2, MTP3, PLE-from-SSD (mannydelrio)","summary":"Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE) in AWQ W4A16 (leoncca g32 build @ fa56146): PP2 with manual layer partition 25,23 (no P2P on CMP 170HX \u2014 PP beats TP, which pays a host-memory sync per layer), MTP 3-token draft, and the 29.8 GB PLE embedding table served FROM NVMe SSD via tinybeast plugins (TB_PLE_SSD=1 + prefetch so SSD reads overlap the previous GPU step). 4\u20135.7x faster than the stock TP baseline on the same cards at every prompt size, full native 262,144-token context, 897K tokens of KV. Measured by the author: 109/159 tok/s answers prose/code at 1 request (was ~37), 283/416 aggregate at 4 concurrent, prompt reading ~7,700 tok/s, TTFT 13.0 s at 100K. Quality unchanged: HumanEval 155/164, 8/8 long-doc to 222K. Engine is a SOURCE BUILD: wtdcode/vllm-backport @ 29e66dad + the repo's patches \u2014 no published image, runtime notes carry the build steps. Cards are 8GB-variant CMP 170HX unlocked to 64 GB by cmpunlocker v0.5; 180 W/card.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":109,"prefill_tok_s":7700,"updated_at":"2026-10-31 11:51:34.514899-06","commit_sha":"744dc3943504b5846fbdf2200bb5b74dd00ca990","content_hash":"902bcf143162651b991af84f818622b06504a8ab","votes":0,"credit":[{"name":"mannydelrio","url":"https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx"},{"name":"cmpunlocker (amoghmunikote)","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"clee/vllm-sm80","url":"https://github.com/clee/vllm-sm80"},{"name":"wtdcode (vllm-backport)","url":"https://github.com/wtdcode/vllm-backport"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3.json","alternate":"/r/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3"}},{"slug":"qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots","title":"Qwen3.8-Flash-Next-Uncensored AWQ W4A16 on 2x CMP 170HX \u2014 8 slots, PP2, MTP5 + int4 draft head, PLE-from-SSD (mannydelrio)","summary":"Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE), AWQ W4A16 g32: PP2 (CMP 170HX has no P2P), MTP 5-token drafting with an int4 draft-only lm_head, 8 slots, and the 29.8 GB PLE table on NVMe. Single user 240 / 225 / 156 tok/s structured / code / prose; 8 users 1,044 / 715 / 625 aggregate; cold prefill ~6,960 tok/s; 262,144 context; HumanEval 157/164. 2x CMP 170HX unlocked to 64 GB, Gen2 x4, 180 W each.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":240.400000000000005684341886080801486968994140625,"prefill_tok_s":6964,"updated_at":"2026-10-31 11:51:33.210587-06","commit_sha":"545d29f3458cc8a926b187626f9ba5fb06c6c84f","content_hash":"abb0c007d26314948f3b53f90f638ba494e38895","votes":0,"credit":[{"name":"mannydelrio","url":"https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx"},{"name":"AlexBrandes (W4A16 draft lm_head patch, quantizer)","url":"https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next"},{"name":"cmpunlocker (amoghmunikote)","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"clee/vllm-sm80","url":"https://github.com/clee/vllm-sm80"},{"name":"wtdcode (vllm-backport)","url":"https://github.com/wtdcode/vllm-backport"}],"uploaded_by":"mannydelrio1","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots.json","alternate":"/r/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots"}},{"slug":"glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash max effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records max reasoning effort, selected per request with model GLM-5.3-Flash-max. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":41.5,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:37.835428-06","commit_sha":"663174ab27ccbc0f93296953b897b19e08b64726","content_hash":"1bf0157fd231d61ac68175f7a1493df8e9583746","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash low effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records low reasoning effort, selected per request with model GLM-5.3-Flash-low. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":41.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:37.671135-06","commit_sha":"7a55561ae92fcdeb4a011950b04c4848b7666861","content_hash":"1c7b1a323fae222fd633e500b576c9f1fe1e7cec","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash high effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records high reasoning effort, selected per request with model GLM-5.3-Flash-high. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":43.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:26.881493-06","commit_sha":"26ce83d34683309b83e4478edb3ed1213018d87f","content_hash":"58f06fe95f6c37780466d207181753f0f7c5b12f","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3","title":"Qwen3.8-Flash-Next W4A16 INT4PLE on 2x modded CMP 170HX \u2014 PP2 MTP3 (DUFRelic)","summary":"Production 2x CMP 170HX recipe for Qwen3.8-Flash-Next: zankich INT4PLE repackage of halt95 Merlin W4A16 (94.4 GiB, 51B-param PLE table repacked INT4) on stock vLLM 0.30.0 + a 6-layer hash-guarded overlay stack (PP2/PLE #56444, KV filter #54793, MTP-over-PP #54709, slot statefix #55506, SpeedSuite 5-PR overlay, batched-P2P). PP2/TP1, MTP 3-token draft (measured 52.27% accept). Hardware is the recipe's own mod stack: 74-SM unlock (cmpunlocker), VBIOS 92.00.6D.00.0A, 170tune 1420 MHz SM cap + HBM 1728 MHz undervolt, BAR1 P2P driver patches, 140 W/card. Author-measured 2026-10-03 (median of 3, isolation audit): prefill C1 6213.7 tok/s @16k, decode C1 133.4 tok/s (SD 22.8 reported), decode C4 agg 245.2 tok/s; 55/56 C at 160 W peaks, no Xid/OOM. INT4 PLE table (~26.8 GiB) pinned in HOST RAM; KV BF16 (pinned build's QSA path rejects FP8). Driver mods optional for function but required for the published throughput.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":133.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:28.370851-06","commit_sha":"c26a7fd62ded418a43f5a10abf82eaedb7afc6bc","content_hash":"7c6051a80e2a07b8bca384eb300935641c88349b","votes":0,"credit":[{"name":"DUFRelic","url":"https://github.com/DUFRelic/qwen38-flash-next-recipe"},{"name":"zankich (INT4-PLE repackage + plugin)","url":"https://github.com/zankich/vllm"},{"name":"halt95 (W4A16-Merlin base checkpoint)","url":"https://github.com/halt95/merlin-engine"},{"name":"michael-lgtm (public overlay rebuild)","url":"https://github.com/michael-lgtm/qwen38-flash-next-cmp-overlay"},{"name":"amoghmunikote (cmpunlocker SM unlock + BAR1 P2P)","url":"https://github.com/amoghmunikote/cmpunlocker"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3.json","alternate":"/r/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3"}},{"slug":"qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6","title":"Qwen3.8-Flash-Next NVFP4 2x CMP 170HX TEP2 GDS SSD PLE MTP6","summary":"Recipe by nguyenthimy2022kg-alt. Community patched vLLM run of the NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on 2x unlocked CMP 170HX (SM80, ~63.4 GiB usable each) with TEP2 tensor+expert parallel and fixed MTP6 speculative decoding with per-request draft isolation. v0.3.0 supports 1-10 active requests (max_num_seqs accepts 1-10, defaults to 1 when omitted; repo example sets 10) with compact/topk vocabulary transport and expanded PCIe communication; history-copy (ngram hybrid) drafting removed. The ~51.2 GB FP8 PLE table stays on NVMe, read GPU-direct via GDS/cuFile with 32 persistent readers and 3 staging slots - no CPU-memory relay; optional host-read PLE backend off by default. FlashInfer PCIe IPC AllReduce, 36-token prefix matching, bayley/cmpunlocker BAR1/P2P driver + host GDS. Measured upstream (2026-10-11, thinking off): 10 requests x 5,000 tokens, 749.67 tok/s aggregate, 76.29/request; historical 2026-09-08 window 151.05-168.29 tok/s decode at 8K-128K inputs xhigh thinking.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":158.19999999999998863131622783839702606201171875,"prefill_tok_s":2059.90000000000009094947017729282379150390625,"updated_at":"2026-10-29 21:32:27.639988-06","commit_sha":"7d911c7dd2ce6b0a45d3c14791ee74036a43e7ae","content_hash":"247ee93d0bf55ac542edef7091844ac3df34a323","votes":0,"credit":[{"name":"nguyenthimy2022kg-alt","url":"https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6.json","alternate":"/r/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6"}},{"slug":"qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm","title":"Qwen3.8-Flash-Next 3bpw FP8-PLE 1x CMP 170HX DFlash2 RecoverSSM","summary":"Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s editing code, 799.5/791.3/586.4 across eight, up to 48 requests at once - 1,384 tok/s total with the default DFlash2 drafter (1,403 with the checkpoint's MTP head), 4,243 tok/s cold prefill, 292,784-token KV pool. Serves klee100's 3 bpw AutoRound checkpoint (3-bit g128 / 2-bit g64 experts, 47 GiB on card) with halt95's FP8 n-gram table in HOST RAM (48 GiB, hard-linked at build) on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) so the whole decode step is one CUDA graph on a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3 by request count) + the recipe's RecoverSSM + BF16 GDN-state patches (9 files mounted over the image) fit 48 concurrent requests on one card. Measured 2026-10-23 under the MiaAI-Lab protocol.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":197.80000000000001136868377216160297393798828125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.955646-06","commit_sha":"af7d7dde12376551476c8b9e36cd72c68dc32408","content_hash":"c82e80ffa9955d905b61bda381cf7d89db1350ba","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/Mia-AiLab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json","alternate":"/r/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX TP2 DFlash2 K7 hybrid drafting 393K (alexvnesta)","summary":"Tensor-parallel successor to Flun's 2x CMP 170HX layer-split recipe: ExLlamaV3 TP2 over both 64 GB cards with a 655,360-token Q8 KV pool (392,960-token request cap), DFlash2 K7 speculative decoding plus a NEW hybrid drafting mode (H=15 draft tokens, temperature-0 outputs byte-identical on/off), two concurrent sessions with fair scheduling, idle sessions parked in system RAM (resume 1-3 s), a cross-session 26.8K-prefix cache (repeat TTFT 20.8 s -> 0.7 s), and image input (base64 data URLs, bwrap-sandboxed decode). TP backend NCCL 2.30.7 over GPU P2P after content-verified p2p_check; beats the native host-RAM backend by +4.35/+6.48 % decode and +10 % prefill (serial n=10); TP_BACKEND=native is the fallback. Author A/B on one machine (2026-10-12): 123.6 / 44.3 tok/s easy/hard text vs Flun layer-split 96.2 / 34.2 on the SAME box (+28 %), hybrid drafting 134-147 tok/s on code-edit/JSON. OpenAI-compatible 127.0.0.1:8012/v1, no auth (loopback/proxy). STOCK cards only: PCIe Gen2 x16, 180 W.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":123.599999999999994315658113919198513031005859375,"prefill_tok_s":1291.5,"updated_at":"2026-10-29 21:32:25.153466-06","commit_sha":"4a4d9f8d79ff001c6c845746708a86e4a3025350","content_hash":"32bd428f49f6a62760cd5c983a7324bfd38163eb","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-tp2-dflash2-k7-hybrid"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x CMP 170HX \u2014 DFlash2 K7 Q8 KV 384K (Flun)","summary":"First ExLlamaV3 serving recipe on 2x CMP 170HX: full-VRAM HBM-resident GLM-5.3-Flash EXL3 3.05bpw (~116.6 GiB across 2x64GB, no CPU/SSD tier) with DFlash2 K7 speculative decoding \u2014 drafter re-quantized locally BF16->EXL3 6bpw (~0.96 GiB) with an 8K-token GPU ring KV cache so a 384K request needs no 384K draft cache (bit-exact over ring wraps). Q8 quantized MLA latent KV, k_hcfuse fused decode kernels (SM80 JIT, shadow-check bit-exact), process-local monkeypatches only (engine tree unmodified), opt-in previous-request DFlash prefix reuse (8K repeat TTFT 7.9s->1.1s; general follow-up defect documented, not fixed), OpenAI-compatible API with native PP/TG + acceptance timings. Alternative stock MTP d2 profile (fp16 192K). Author-reported ~90 tok/s sustained, 110-120 at high acceptance (Reddit 1x0b1ws; no run file). Cards cap-modded PCIe x16 (Gen2 x8 vs stock x4), no P2P; single-request; Q8 changes numerics.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":90,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:24.781602-06","commit_sha":"f363025763f08eecd079592032038eb42acf00ae","content_hash":"f3d955e45e537ed75550188c286a6fed02089829","votes":1,"credit":[{"name":"Flun (recipe, stack, profiles, drafter conversion)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused decode kernels)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"MiaAI-Lab (GLM-5.3-Flash-EXL3-2x-DGX-Sparks \u2014 conversion reference)","url":"https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-cmp-170hx-dflash2-k7-q8-kv-384k"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin","title":"GLM-5.3-Flash EXL3 3.05bpw on 1x RTX 3090 \u2014 RAM/NVMe expert tiering (near-exact)","summary":"320B MoE served on ONE 24 GB RTX 3090 with stock exllamav3 1.5.1 plus process-local monkeypatches: the 12,096 routed experts (42 MoE layers x 288, 8 picked per token per layer) tier across VRAM, pinned host RAM and a packed NVMe store. An elastic VRAM cache holds the ~1,300 hottest experts (12.4 GB); evicted experts move into a pinned no-duplicates RAM tier sized from the container cap; a 117 GB O_DIRECT record store covers the rest, with layer-ahead prefetch, a victim write-back ring and an AVX2 CPU lane computing RAM-resident decode picks in place. Modes: fast (all experts in RAM, ~218 GiB free, 28.2 tok/s, KL 0.005), nvme (55 GiB cap, 19.5 tok/s, KL 0.008; 16 GiB cap 14.7 lab / 14.38 sweep), nvme-exact (bit-exact, 8.3 tok/s), and an experimental Intel Arc Pro B70 peer-offload tier (GLM53_B70=1, 26.5 tok/s, KL 0.003 - not lab-run). 131,072-token KV, OpenAI API on port 30000, model id glm-5.3-flash, no output caps. Registry lab six gates pass at 55 GiB (PR #199); 16 GiB reported.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":24.10000000000000142108547152020037174224853515625,"prefill_tok_s":1015,"updated_at":"2026-10-29 21:32:24.549403-06","commit_sha":"53520bbf5e06f65db6670e8693efb018f75026c8","content_hash":"65ab07a0787590e7c21331b006d4a13553080d02","votes":0,"credit":[{"name":"sybil-solutions (image, tiering runtime, omarchy plugin)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"turboderp (ExLlamaV3 + EXL3 3.05bpw checkpoint)","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin"}}]}