{"meta":{"count":14,"page":1,"pages":2,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["170hx-x4","rtx-3090-x2"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T15:41:28+00:00"},"recipes":[{"slug":"deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake","title":"DeepSeek-V4.1-Flash-1017 Hybrid EXL3 TP4 4x CMP 170HX Morrowmake","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.0 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw export, every other weight from DeepSeek's official FP8 1017 checkpoint (sha256-validated link build, no copies), and the model's ~189 GiB Engram tables in host RAM (VLLM_ENGRAM_PLACEMENT=auto, SSD fallback below ~212 GiB available). up to 319 tok/s one user structured (287 code / 162 prose), up to 765 across eight, up to 3,818 tok/s cold 2K prefill and a 1,247,406-token KV pool at 262,144 context, full-precision BF16 KV. Engine: Morrowmake's vLLM fork @ a0c57b52 on upstream 4e0681ed56 (sm_80 EXL3 + Marlin kernels, custom P2P all-reduce after a startup content check, DSpark speculative decoding 7/5 adaptive), pinned container image, TP4 with whole routed experts per card, optimised for PCIe x16 links. HumanEval 157/164, GSM8K 1,280/1,319, within test noise of the unmodified-checkpoint reference.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":162,"prefill_tok_s":3804,"updated_at":"2026-10-30 08:13:13.515043-06","commit_sha":"1d43d37b0b9cffa384845e62bcd3af425c04088c","content_hash":"79c2a1c9569e13d38af257c3a7c84e011e8a9cbe","votes":0,"credit":[{"name":"Morrowmake (recipe, engine fork, measurements)","url":"https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe"},{"name":"LibertAI - EXL3 2.9-bpw routed-expert quantization","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"cmpunlocker (amoghmunikote) - P2P driver basis + minimal TRAP31 patch","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"MiaAI-Lab - decode benchmark protocol (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"Marlin (IST-DASLab + Neural Magic contributors) - kernel basis","url":"https://github.com/IST-DASLab/marlin"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake.json","alternate":"/r/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake"}},{"slug":"glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141","title":"GLM-5.3-Flash on 4x CMP 170HX \u2014 Morrowmake recipe 1.8.0, TP4 DFlash2","summary":"PP4 (pipeline-parallel 4, TP1) as-run on a 4x CMP 170HX rig 2026-09-28 at --gpu-memory-utilization 0.93 after 0.95 OOM-killed warmup (a 72 MiB KDA prefill workspace did not fit in the ~70 MiB left on the last card). Author 1.8.0 measurements below are TP4 unless labeled PP4; KV pool 2,088,960 tokens under PP4 (author-published 1.8.0 figure). Recipe release 1.8.0 2026-10-16 (engine 819146023f, image engine-819146023f) \u2014 updated from 1.7.2; engine rebased on upstream vLLM 4e0681ed56, same context/batch/fair-prefill defaults; 21 engine-switch env names renamed (old names silently ignored at code defaults); container-only install; author re-measured throughput/KV tables at 1.8.0.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":264.3999999999999772626324556767940521240234375,"prefill_tok_s":6264,"updated_at":"2026-10-29 23:29:37.313564-06","commit_sha":"6e47f035b02d04cee06466adb7fb69b21d9d5240","content_hash":"171456f9f739f5c140a9a08da526b95e056ea9fe","votes":1,"credit":[{"name":"Morrowmake","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141.json","alternate":"/r/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141"}},{"slug":"deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81","title":"DeepSeek-V4.1-Flash-1017 Fidelity-First 4x CMP 170HX alanoo81","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on 4x CMP 170HX (sm_80, 64 GB each, PCIe Gen2 x16), TP4, as a fidelity-first variant of the GNmac-lab recipe 1.2.0 @ 2ad5741ab1 with four changes: routed experts of layers 18-22 grafted 2-bit to 3-bit (LibertAIDAI EXL3 3.5bpw shards 00020-00024-of-00044 into the 2.9bpw pack, all 40 layers 3-bit); no DSpark drafter (SPEC_TOKENS=0, frees ~2.8 GiB/card for KV); Engram tables genuinely loaded (ENGRAM_STORAGE=ssd from a 200 GiB RAM tmpfs); gpu_memory_utilization 0.98, KV pool 2,717,420 tokens at 262,144 context. Author-measured: 300/300 exact identifier copies, byte-identical repeats; perplexity -1.80% overall / -4.45% French vs the 2.9bpw pack (95% CIs exclude zero); ~105 tok/s decode c1, 449-560 tok/s aggregate c8; cold prefill 2,514 tok/s median. Documents a silent failure: leaving DSV41_ENGRAM_STORAGE unset (host mode) leaves the Engram tables unfilled - layers 1 and 14 act as identity functions, 15/20 copies, temp-0 output drift, no error.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":105,"prefill_tok_s":2514,"updated_at":"2026-10-29 21:57:43.459028-06","commit_sha":"5af1460f4a2dce4f590f6fdfcd108119285e3e7c","content_hash":"67eb1810e7eea071e85e8edab747fd98247dd872","votes":0,"credit":[{"name":"alanoo81 (recipe, graft, measurements)","url":"https://github.com/alanoo81/deepseek-v41-flash-4x-cmp170hx"},{"name":"GNmac-lab (base recipe 1.2.0 @ 2ad5741ab1, MIT)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"LibertAIDAI (EXL3 2.9bpw @ 0be63c33 / 3.5bpw @ 04a492fc)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"Curtis White (hybrid layout + quantization_config.json)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"vcruz305/vllm-exl3 (AGPL-3.0-only, inside the built image)","url":"https://github.com/vcruz305/vllm-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81.json","alternate":"/r/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81"}},{"slug":"deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash EXL3 3bpw TC-MoE on 4x CMP 170HX TP4 (original checkpoint)","summary":"DeepSeek-V4.1-Flash (ORIGINAL deepseek-ai/DeepSeek-V4.1-Flash checkpoint, quantized by Mia-AiLab as EXL3 3.02bpw \u2014 NOT the 2026-10-17 '1017' re-release, which has its own sibling recipe) served EXL3 3bpw tensor-parallel 4 on four CMP 170HX cards (sm_80, 140 W, HBM locked 1728 MHz) in a 2x Xeon E5-2689 v4 Broadwell host, PCIe Gen2 x16, no NVLink, using Morrowmake's tensor-core MoE kernels + host-staged all-reduce ported onto this checkpoint: K3 on every served layer with a split K2 down-projection, row-rule hybrid (coop rows 1-42, moe_out_v2 43-64, moe_out_prefill 65-2048), one shared 540.4 MiB MoE workspace per GPU, zero-register-spill served kernels. Measured by the author: 178.3 tok/s structured decode c1, 587 tok/s across 8 users, median TTFT 8.37 s from 2K through 255K context (10.9 s at 255,424) against the same host's optimized graphs+coop+NCCL stack at 172.7 / 456 / 10.15 s; needles 9/9 through 256K; KV pool 616,916 tokens.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":149.19999999999998863131622783839702606201171875,"prefill_tok_s":1486,"updated_at":"2026-10-29 21:32:23.373585-06","commit_sha":"dc2dbe7cce66fb0f4d3dfad6b6765ed3c4ea8f74","content_hash":"5640bd33a3ca7d6a71e5a4804e0ec780e0c34fb1","votes":0,"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-exl3-3bpw-tc-moe-on-4x-cmp-170hx-tp4"}},{"slug":"deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4","title":"DeepSeek-V4.1-Flash-1017 EXL3 2.9bpw TC-MoE on 4x CMP 170HX TP4","summary":"DeepSeek-V4.1-Flash-1017 (the 2026-10-17 re-release checkpoint, NOT the original DeepSeek-V4.1-Flash) as a hybrid quant on the TC-MoE port: official deepseek-ai/DeepSeek-V4.1-Flash-1017 FP8 tensors for attention, shared experts, head, DSpark draft and Engram (shards 47/48), with LibertAIDAI/...1017-EXL3-2.9bpw routed experts (layers 0-39, mostly 3-bit, layers 18-22 2-bit) TP4 on four CMP 170HX cards (sm_80, 140 W, ~62.5 GB/card). Same Morrowmake kernels + image as the EXL3 3bpw TC-MoE recipe; VLLM_EXL3_COOP=1; EXL3 arena cache (~1.1 GiB/layer, load shortcut only). 1017 quantization_config.json is the switch (dsv41_main_routed_experts_only, mtp_experts=source, non-routed stays FP8, compressor projection BF16). Author-measured: 180.4/137.1/96.6 tok/s structured/code/prose c1; 8-user aggregate 621.1/376.1/282.5 vs 564.7 previous Mia 3.0bpw. Cold 131k prompt 1,330 prefill / 163.9 decode. Needles 9/9. KV pool 1,534,065 tokens; vision on, drafter text-only.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":165.599999999999994315658113919198513031005859375,"prefill_tok_s":1390.09999999999990905052982270717620849609375,"updated_at":"2026-10-29 21:32:22.449013-06","commit_sha":"306225cea85cf158c3036c971b031a9c1740075b","content_hash":"59b7b30fa1a9d7fe2197055fbfc51c994b611ccc","votes":0,"credit":[{"name":"CurtisAccelerate","url":"https://github.com/CurtisAccelerate"},{"name":"Morrowmake","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"Measured serve used local image sha256:55d2bd9a6664 (no published image)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"turboderp-org/exllamav3 - EXL3 codec @ d3739fd3 (MIT; Copyright (c) 2025 Turboderp)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"PixelML/sm80vllm branch sm80 (DeepseekV4AmpereMLAAttention) - SM80 attention for this model","url":"https://github.com/PixelML/sm80vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4.json","alternate":"/r/deepseek-v4-1-flash-1017-exl3-2-9bpw-tc-moe-on-4x-cmp-170hx-tp4"}},{"slug":"deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx","title":"DeepSeek-V4.1-Flash-1017 Engram-NVMe TP4/PP4 4x CMP 170HX","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards power-capped at 180 W, with the model's 189 GiB Engram n-gram tables streamed from NVMe on demand. Recipe 1.2.0 (author measurements, claimed_by=submitter T0): 355.7/258.4/183.6 tok/s one user (structured/code/prose), 867.6/544.0/440.5 across eight, 2,416 tok/s cold prefill (TP4; PP4 2,311), 1,395,731-token KV pool at 262,144 context (1,270,789 at 1M), HumanEval 154/164 (6 length-capped; McNemar p>=0.5 vs 156/164), GSM8K 1,275/1,319 (TP4). Hybrid checkpoint assembled without copying tensors: LibertAI EXL3 2.9-bpw routed experts (layers 0-39) + official FP8 attention/shared experts/head/DSpark drafter/Engram shards + BF16 embeddings/LM head/vision, fp8_ds_mla KV. Engine: vLLM on the community sm_80 stack (zebgop-ops sm80 sparse attention + DSpark V1 + Engram row store, FlashInfer PCIe IPC all-reduce) plus this recipe's measured keep-or-drop patches. TP4 (default, needs x16 + P2P) and PP4 (x1/x4, no P2P) from one .env switch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":150.599999999999994315658113919198513031005859375,"prefill_tok_s":0,"updated_at":"2026-10-28 23:26:24.539707-06","commit_sha":"1b9a60d70c9e1e1f83c7db45556ac3450d511600","content_hash":"dadcb54c6118ad769c07ea0930690a53cead0c65","votes":0,"credit":[{"name":"GNmac-lab / greencandle (recipe, patches, measurements)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"CurtisAccelerate - TP4 hybrid-checkpoint recipe + tensor-core EXL3 MoE port","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-EXL3-TC-MoE"},{"name":"zebgop-ops - dsv41reap-pp sm_80 sparse attention, PP shadow KV, DSpark V1, Engram row store","url":"https://github.com/zebgop-ops/dsv41reap-pp"},{"name":"vcruz305 - vllm-exl3 (AGPL-3.0-only)","url":"https://github.com/vcruz305/vllm-exl3"},{"name":"LibertAI - EXL3 2.9-bpw quantization (MIT)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"kaka86mm - dsv41-flash-pp4-170hx (watched sibling)","url":"https://github.com/kaka86mm/dsv41-flash-pp4-170hx"},{"name":"Morrowmake - CMP 170HX recipe format, P2P work, tensor-core MoE kernels","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"Mia-AiLab - decode benchmark protocol + prompts (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"FlashInfer - PCIe IPC all-reduce","url":"https://github.com/flashinfer-ai/flashinfer"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx.json","alternate":"/r/deepseek-v4-1-flash-1017-engram-nvme-tp4-pp4-4x-cmp-170hx"}},{"slug":"deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram","title":"DeepSeek-V4.1-Flash-1017 dsx 2.6-bit GPTQ 2x RTX 3090 (Engram)","summary":"DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) quantized to 187.6 GB single GGUF (2.6127 bpw routed-expert average: 5717 PTQ1_0 / 2545 Q2_0 / 7098 Q3_K per-expert GPTQ with allocator-spent bit budget; dense Q8_0 attention/shared expert/head, F16 embedding/compressor/indexer/Engram-wkv, F32 norms/router/mHC), served by dsx, a Rust+CUDA engine built for this checkpoint, on two RTX 3090s with 192 GB RAM. ~26 tok/s decode / ~170 tok/s prefill at --ctx 262144 with all 15,360 routed experts resident. Exact routing: no expert pruning, no router bias - the rank file orders placement only. Engram n-gram tables (~203 GB) are NOT quantized: read at runtime from the official safetensor shards by a prefetch thread. Hottest ~3,300 experts in VRAM across both GPUs (GPU0's set adapts to the conversation, DSX_ADAPT=8 swaps/token), remaining experts computed on 14 CPU threads from a hugepage RAM arena. Quality vs FP8/FP4 teacher (dsx eval, Lucebox 1017 set): KL 0.1972, top-1 82.59%, PPL 9.3626.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":26,"prefill_tok_s":170,"updated_at":"2026-10-28 15:52:50.29183-06","commit_sha":"942f2d4b4ca3b3ca6697019daf09d9615a47b0af","content_hash":"57641e58a944d825b6d88744c42b9451d8ee916a","votes":0,"credit":[{"name":"lxe (dsx engine + 2bit reference quant)","url":"https://github.com/lxe/dsx"},{"name":"icecubetr (2.6-bit GPTQ quant + measurements)","url":"https://huggingface.co/icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit"},{"name":"deepseek-ai (DeepSeek-V4.1-Flash-1017 checkpoint)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"Lucebox (1017 eval set)","url":"https://huggingface.co/Lucebox/DeepSeek-V4.1-Flash-1017-ROCMFP-GGUF"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram.json","alternate":"/r/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram"}},{"slug":"qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy","title":"Qwen3.8-Flash-Next-FP8 4x CMP 170HX \u2014 PP4 MTP-4 patched vLLM (g4bwy)","summary":"Recipe by g4bwy. Official Qwen/Qwen3.8-Flash-Next-FP8 on 4x unlocked CMP 170HX (GA100 sm_80, 74 SMs, 64 GB) with PP=4 (layer partition 13,12,12,11) and MTP-4 speculative decoding on pinned vLLM mainline 183625d4f9 (2026-10-13, precompiled wheel 0.32.0rc1.dev34+g183625d4f) plus a 7-patch series applied with git am (tree hash must equal 18e0ec8942f61f82fd3a13bd9d9c6b9ce67ad371; 2 patches adopted from upstream PRs #56444/#58094) that unblocks PP+MTP+PLE on sm_80. 1M context at static YaRN 4.0, PLE n-gram table in pinned host RAM (--engram-config cpu_offload), Mamba align prefix caching with retention 16160 + match-unit 16 (99.8-99.9% hits), NCCL P2P disabled (host-staged wins at PP=4), expandable_segments, 200 W per-card caps, uv venv, no docker. Optional W8A16 int8-dense lane (checkpoint/) +15-20% single-stream.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":183.19999999999998863131622783839702606201171875,"prefill_tok_s":10038,"updated_at":"2026-10-20 21:52:36.721138-06","commit_sha":"540d404493ab67413a71efa0f5e541d18d329313","content_hash":"ab27af4ddefebc602e00353e6d7b640e515f9c4d","votes":0,"credit":[{"name":"g4bwy","url":"https://github.com/g4bwy/qwen3.8-flash-next-cmp-170hx"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy.json","alternate":"/r/qwen3-8-flash-next-fp8-4x-cmp-170hx-pp4-mtp-4-patched-vllm-g4bwy"}},{"slug":"qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4","title":"Qwen3.8-27B AWQ-INT4 + DSpark on RTX 3090s \u2014 SGLang TP1/TP2/TP4","summary":"Validated SGLang serving for the dense Qwen3.8-27B hybrid GDN vision-language model (48 Gated-DeltaNet + 16 gated-attention layers) on RTX 3090 24 GB: one digest-pinned container whose TP env selects tuned 1x/2x/4x presets. AWQ-INT4 W4A16 (Marlin) beats FP8 on Ampere; DSpark speculative decoding (RadixArk 1.36B draft) on 2x/4x with auto-fallback on TP1. Image input everywhere, video on 2x/4x. Author-measured: 136.8 tok/s decode at 8k on 4x (169.8 aggregate at C8), 112.9 on 2x, 43.2 on 1x; up to 128k context on 4x.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":127.099999999999994315658113919198513031005859375,"prefill_tok_s":1372.700000000000045474735088646411895751953125,"updated_at":"2026-10-20 21:52:36.534482-06","commit_sha":"26a1363966d9065a310e9ec7caf124206a25f3f2","content_hash":"3fa45739100d87065553d5fb598424a66d4742fb","votes":0,"credit":[{"name":"sybil-solutions / 0xSero (SGLang presets, image build, benchmarks)","url":"https://github.com/sybil-solutions/qwen-serve"},{"name":"cyankiwi (AWQ-INT4 W4A16 checkpoint)","url":"https://huggingface.co/cyankiwi/Qwen3.8-27B-AWQ-INT4"},{"name":"RadixArk (DSpark speculative drafter)","url":"https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4.json","alternate":"/r/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x RTX 3090 \u2014 vLLM + ExLlamaV3 CPU expert offload, 256K (DominikBucko)","summary":"GLM-5.3-Flash (turboderp EXL3 3.05bpw) tensor-parallel on 2x RTX 3090 24 GB + 128 GB system RAM: vendor vLLM image with the spele1100/vllm-glm53 GLM-5.3 overlay, ExLlamaV3 @ 499890c with EXL3 MoE decode kernels, and CPU expert-offload pools (UVA exact host registers, per-layer LRU expert cache with the hottest 20% pinned, glm53_host.py pool planner, checkpoint lock + k-pool index installer). Bounded 4,096-token thinking budget + preserved reasoning for agent loops, 262,144-token context, prefix caching, OpenAI-compatible API. Docker image + systemd example, reproducible from repro.lock.json pins; measured up to 1,155 tok/s prefill and 47-50 tok/s decode with needles-at-32K/131K/260K + coherence gates passing.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":49.39999999999999857891452847979962825775146484375,"prefill_tok_s":1130.5,"updated_at":"2026-10-20 21:52:36.396387-06","commit_sha":"c93c82b83e6832d41dd10a7037c607a3f4b53d1b","content_hash":"c807eb1b13116b97fedcab43cef2aad9dca165e7","votes":0,"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/glm53-flash-2x3090"},{"name":"turboderp-org (EXL3 quant + ExLlamaV3)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"spele1100 (GLM-5.3 vLLM overlay)","url":"https://github.com/spele1100/vllm-glm53"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp"}},{"slug":"mimo-v2-6-flash-rl-309b-moe-nvfp4-4x-cmp-170hx-tp4-dflash-1m-ctx","title":"MiMo-V2.6-Flash-RL 309B MoE NVFP4 4x CMP 170HX TP4 DFlash 1M ctx (ElectroHobo)","summary":"XiaomiMiMo's MiMo-V2.6-Flash-RL (309B MoE, 15B active, text/image/video/audio input) on four CMP 170HX cards (sm_80) at the full 1,048,576-token context: 322/275/170 tok/s one user (structured/code/prose), 754/738/556 across eight, 2,720 tok/s cold prefill, 1,615,953-token KV pool. NVFP4 weights (ModelOpt mixed precision via Marlin, bf16 activations), bf16 KV, MiMo's own DFlash drafter (7 drafts one user, 7/5/3 by request count), full-decode-step CUDA graph, one engine file bind-mounted (upstream vLLM #58235 vision-sink fix), every weight on the GPUs. Morrowmake's first listed community recipe on vllm-cmp170hx; measured 2026-10-18 under the Morrowmake bench protocol.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":274.8999999999999772626324556767940521240234375,"prefill_tok_s":0,"updated_at":"2026-10-20 15:34:18.178415-06","commit_sha":"12a7bfc3d0f95506c73c0dbaf6bb13286ef57281","content_hash":"6388d1808174e278654541bfad590d93a9202607","votes":0,"credit":[{"name":"ElectroHobo (Applied Integrators)","url":"https://github.com/ElectroHobo/mimo-v26-flash-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx fork + engine image + start.sh script set, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"XiaomiMiMo (MiMo-V2.6-Flash-RL base model)","url":"https://huggingface.co/XiaomiMiMo"},{"name":"ProCreations (NVFP4 checkpoint)","url":"https://huggingface.co/ProCreations/MiMo-V2.6-Flash-RL-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/mimo-v2-6-flash-rl-309b-moe-nvfp4-4x-cmp-170hx-tp4-dflash-1m-ctx.json","alternate":"/r/mimo-v2-6-flash-rl-309b-moe-nvfp4-4x-cmp-170hx-tp4-dflash-1m-ctx"}},{"slug":"qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa","title":"Qwen3.8-Flash-Next W4A16-FP8PLE on 2x RTX 3090 \u2014 TP2, expert-offload, MTP3 (DominikBucko)","summary":"Two-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in the albucino W4A16-FP8PLE build (Intel AutoRound INT4 backbone + FP8 PLE table + INT4 MTP draft, checkpoint @ ef55414): TP2 + expert parallel on the humming MoE path, UVA CPU offload (30 GB of cold experts in RAM), dynamic LRU expert cache (hot-84), BF16 KV at full 262,144-token context, MTP 3-token speculative draft, prefix caching. Author-measured (submitter T0, 128 GB RAM host): 131K prompt first token in 31.3 s and up to 4,191 tok/s prefill on the 8K-chunk prefill profile; agent 128K profile decodes 104.8-111.5 tok/s; fast 256K profile reads 260,096 tokens in 90.8 s (2,865 tok/s). Also ships 64 GB RAM and opt-in uncensored (abliteration) profiles. Engine is the pinned vLLM runtime image 0.1.dev20073+g8e685d198, republished as a release image pinned by digest; argv transcribed verbatim from scripts/serve-container.sh @ 939905b.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":106.5,"prefill_tok_s":2947,"updated_at":"2026-10-20 15:34:08.646082-06","commit_sha":"bb7322bf4f7bdf564d617b1ab9f9d2985bf71877","content_hash":"5e0fd674c07293431c87af75dcd971af50a99d37","votes":0,"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/qwen38-flash-next-2x3090"},{"name":"albucino (checkpoint)","url":"https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE"},{"name":"Intel AutoRound (W4A16 target)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"RadixArk (FP8 PLE table)","url":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa.json","alternate":"/r/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa"}}]}