{"meta":{"count":31,"page":1,"pages":3,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":null,"generation":null,"task":"vision","tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T03:28:16+00:00"},"recipes":[{"slug":"deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake","title":"DeepSeek-V4.1-Flash-1017 Hybrid EXL3 TP4 4x CMP 170HX Morrowmake","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on four CMP 170HX cards (sm_80): Morrowmake's release 1.0.0 recipe serves a hybrid checkpoint - routed experts from LibertAI's EXL3 2.9-bpw export, every other weight from DeepSeek's official FP8 1017 checkpoint (sha256-validated link build, no copies), and the model's ~189 GiB Engram tables in host RAM (VLLM_ENGRAM_PLACEMENT=auto, SSD fallback below ~212 GiB available). up to 319 tok/s one user structured (287 code / 162 prose), up to 765 across eight, up to 3,818 tok/s cold 2K prefill and a 1,247,406-token KV pool at 262,144 context, full-precision BF16 KV. Engine: Morrowmake's vLLM fork @ a0c57b52 on upstream 4e0681ed56 (sm_80 EXL3 + Marlin kernels, custom P2P all-reduce after a startup content check, DSpark speculative decoding 7/5 adaptive), pinned container image, TP4 with whole routed experts per card, optimised for PCIe x16 links. HumanEval 157/164, GSM8K 1,280/1,319, within test noise of the unmodified-checkpoint reference.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":162,"prefill_tok_s":3804,"updated_at":"2026-10-30 08:13:13.515043-06","commit_sha":"1d43d37b0b9cffa384845e62bcd3af425c04088c","content_hash":"79c2a1c9569e13d38af257c3a7c84e011e8a9cbe","votes":0,"credit":[{"name":"Morrowmake (recipe, engine fork, measurements)","url":"https://github.com/Morrowmake/dsv41-flash-cmp170hx-recipe"},{"name":"LibertAI - EXL3 2.9-bpw routed-expert quantization","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"cmpunlocker (amoghmunikote) - P2P driver basis + minimal TRAP31 patch","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"MiaAI-Lab - decode benchmark protocol (used as data)","url":"https://huggingface.co/Mia-AiLab"},{"name":"Marlin (IST-DASLab + Neural Magic contributors) - kernel basis","url":"https://github.com/IST-DASLab/marlin"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake.json","alternate":"/r/deepseek-v4-1-flash-1017-hybrid-exl3-tp4-4x-cmp-170hx-morrowmake"}},{"slug":"qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3","title":"Qwen3.8-Flash-Next NVFP4 on RTX PRO 6000 Blackwell \u2014 TP1 MTP3","summary":"Single-card Blackwell recipe for Qwen3.8-Flash-Next (hybrid GDN/QSA MoE, 48 layers, 512 experts) on one 96 GB RTX PRO 6000: the Community image community image R34 (localinferencelab/vllm, v0.26.1rc0+glm53.r34), TP1 with MTP3 speculative decoding (b12x MoE backend), 256k context. Benchmark-qualified on RTX PRO 6000 Blackwell Max-Q: decode 148 tok/s at ctx-0 (temp 1), prefill 3860 tok/s at 2048 batched tokens. Launch argv transcribed verbatim from the r35 community deployment. Weights: JMNI-Labs/Qwen3.8-Flash-Next-NVFP4-QAD5500-Hybrid \u2014 step-5500 QAD trained tensors recombined with main 7c4f1bc1 MXFP8 attention + NVFP4 MTP experts; same-stack measured +5% MTP-normalized decode, +25% with MTP vs the main checkpoint (evals within noise, MMLU-Pro 66.9 vs 67.2).","tier":"curated","beta":false,"generation":"blackwell","decode_tok_s":137.19999999999998863131622783839702606201171875,"prefill_tok_s":6627.3000000000001818989403545856475830078125,"updated_at":"2026-10-29 23:29:38.177885-06","commit_sha":"28709ecba026aff18371426bc26025e5a4185209","content_hash":"1e37b0f978392ba766c01cfdf9ae9ee3e002ee84","votes":1,"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/rtx6kpro"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json","alternate":"/r/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3"}},{"slug":"glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash max effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records max reasoning effort, selected per request with model GLM-5.3-Flash-max. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":41.5,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:37.835428-06","commit_sha":"663174ab27ccbc0f93296953b897b19e08b64726","content_hash":"1bf0157fd231d61ac68175f7a1493df8e9583746","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash low effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records low reasoning effort, selected per request with model GLM-5.3-Flash-low. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":41.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:37.671135-06","commit_sha":"7a55561ae92fcdeb4a011950b04c4848b7666861","content_hash":"1c7b1a323fae222fd633e500b576c9f1fe1e7cec","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw","title":"GLM-5.3-Flash EXL3 2.05bpw on 2x RTX 5090 - TabbyAPI hot-expert swaps, 262K ctx, vision","summary":"GLM-5.3-Flash (321B MoE, ~18B active) turboderp EXL3 2.05bpw on two RTX 5090 + 64 GB DDR5: TabbyAPI on ExLlamaV3 with CPU-offloaded routed experts (96/layer GPU0 layers 3-22+MTP, 102/layer GPU1 23-44), runtime hot-expert swaps, per-GPU expert split, the DSA indexer's per-token rows in a fixed ring, live /metrics token counters, the expert-exchange clock handed to the first text MoE layer so swaps also run at 2-4 streams, MTP head drafting 1 tok/step single-stream, 262,144-token window on an 8-bit KV page pool (~1.6 GiB across 4), vision ON with tower linears pinned in host RAM. Exact-output: no expert skipped, every draft verified. Measured 2026-10-10 (R969, three fresh boots): c1 score 62.9 tok/s (code 59.9/prose 66.7/chat 68.5/html 64.8/edit 53.9), 2/3/4-stream sums 74.4/92.0/98.0. Nine in-repo overlays on the qwen3.8-flash-next-2x-rtx5090 base image.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":62.89999999999999857891452847979962825775146484375,"prefill_tok_s":2023,"updated_at":"2026-10-29 23:29:37.491895-06","commit_sha":"4dc491797af361a6a0f4cdedde5f7dbeb918082d","content_hash":"66ee933dc865f3362d1e2880b2ef8e330f331b05","votes":0,"credit":[{"name":"adrienbrault (recipe, five patch overlays, measurements)","url":"https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090"},{"name":"zai-org (GLM-5.3-Flash)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"turboderp (EXL3 2.05bpw quantisation; ExLlamaV3 + CPU MoE offload)","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"},{"name":"theroyallab (TabbyAPI)","url":"https://github.com/theroyallab/tabbyAPI"},{"name":"adrienbrault/qwen3.8-flash-next-2x-rtx5090 (base image: ExLlamaV3 dev 5783a93 patch stack + TabbyAPI 53da7919)","url":"https://github.com/adrienbrault/qwen3.8-flash-next-2x-rtx5090"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw.json","alternate":"/r/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw"}},{"slug":"glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141","title":"GLM-5.3-Flash on 4x CMP 170HX \u2014 Morrowmake recipe 1.8.0, TP4 DFlash2","summary":"PP4 (pipeline-parallel 4, TP1) as-run on a 4x CMP 170HX rig 2026-09-28 at --gpu-memory-utilization 0.93 after 0.95 OOM-killed warmup (a 72 MiB KDA prefill workspace did not fit in the ~70 MiB left on the last card). Author 1.8.0 measurements below are TP4 unless labeled PP4; KV pool 2,088,960 tokens under PP4 (author-published 1.8.0 figure). Recipe release 1.8.0 2026-10-16 (engine 819146023f, image engine-819146023f) \u2014 updated from 1.7.2; engine rebased on upstream vLLM 4e0681ed56, same context/batch/fair-prefill defaults; 21 engine-switch env names renamed (old names silently ignored at code defaults); container-only install; author re-measured throughput/KV tables at 1.8.0.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":264.3999999999999772626324556767940521240234375,"prefill_tok_s":6264,"updated_at":"2026-10-29 23:29:37.313564-06","commit_sha":"6e47f035b02d04cee06466adb7fb69b21d9d5240","content_hash":"171456f9f739f5c140a9a08da526b95e056ea9fe","votes":1,"credit":[{"name":"Morrowmake","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141.json","alternate":"/r/glm-5-3-flash-4x170hx-tp4-pp4-morrowmake-141"}},{"slug":"deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell","title":"DeepSeek-V4.1-Flash-1017 Kraken TP3 3x RTX PRO 6000 Blackwell","summary":"DeepSeek-V4.1-Flash-1017 at TP3 on three 96 GB RTX PRO 6000 Blackwell cards with the full 1M-token context, serving DeepSeek's original weights (FP8 dense + MXFP4 routed experts, nothing requantized). Expert layers 20-33 park in host RAM against a 1.5 GiB promotion reserve; once healthy their hottest experts promote back to GPU (hot-first order from 1017 route counts, lossless FlashInfer CSF scale compression). Spreading that fixed budget across 14 layers instead of 6 shallows each layer's cold tail - the author reports 68.0 -> 72.3 engine steps/s at 32K decode (+6.3%), no quality, KV or prefill cost. Engram layer 1 pinned in host RAM, layer 14 streamed from NVMe. DSpark speculative decoding, 5 drafts verified, NVFP4 drafter head. 1,048,576-token window over a 2,185,581-token KV pool, up to 4 sequences. Measured on image 20261019.3: prefill 13,486/12,245 tok/s (32K/128K), 7,193 at 983K; decode 186.7/340.7 c1/c4 at 32K, 179.2/307.1 at 128K, 474 aggregate on 4 shorts, 271.8 coding peak.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":228.5,"prefill_tok_s":11445.5,"updated_at":"2026-10-29 23:29:37.139691-06","commit_sha":"fe33370ee6a18643174519c8b9e956f05e807711","content_hash":"252307b64eed922ec9291880d6efb1938f670372","votes":0,"credit":[{"name":"BoxTops4Coochie (recipe, TP3 port, measurements)","url":"https://github.com/BoxTops4Coochie/RTX-PRO-6000-TP3-Inference-Recipes"},{"name":"infernix - ds41f-tp3 expert-parking design on karmic-kraken","url":"https://github.com/infernix"},{"name":"Local Inference Lab - vLLM karmic-kraken fork + FlashInfer b12x kernels","url":"https://github.com/local-inference-lab"},{"name":"DeepSeek - DeepSeek-V4.1-Flash-1017 base model (MIT)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell.json","alternate":"/r/deepseek-v4-1-flash-1017-kraken-tp3-3x-rtx-pro-6000-blackwell"}},{"slug":"glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash high effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records high reasoning effort, selected per request with model GLM-5.3-Flash-high. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":43.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:26.881493-06","commit_sha":"26ce83d34683309b83e4478edb3ed1213018d87f","content_hash":"58f06fe95f6c37780466d207181753f0f7c5b12f","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81","title":"DeepSeek-V4.1-Flash-1017 Fidelity-First 4x CMP 170HX alanoo81","summary":"DeepSeek-V4.1-Flash-1017 (748B MoE) on 4x CMP 170HX (sm_80, 64 GB each, PCIe Gen2 x16), TP4, as a fidelity-first variant of the GNmac-lab recipe 1.2.0 @ 2ad5741ab1 with four changes: routed experts of layers 18-22 grafted 2-bit to 3-bit (LibertAIDAI EXL3 3.5bpw shards 00020-00024-of-00044 into the 2.9bpw pack, all 40 layers 3-bit); no DSpark drafter (SPEC_TOKENS=0, frees ~2.8 GiB/card for KV); Engram tables genuinely loaded (ENGRAM_STORAGE=ssd from a 200 GiB RAM tmpfs); gpu_memory_utilization 0.98, KV pool 2,717,420 tokens at 262,144 context. Author-measured: 300/300 exact identifier copies, byte-identical repeats; perplexity -1.80% overall / -4.45% French vs the 2.9bpw pack (95% CIs exclude zero); ~105 tok/s decode c1, 449-560 tok/s aggregate c8; cold prefill 2,514 tok/s median. Documents a silent failure: leaving DSV41_ENGRAM_STORAGE unset (host mode) leaves the Engram tables unfilled - layers 1 and 14 act as identity functions, 15/20 copies, temp-0 output drift, no error.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":105,"prefill_tok_s":2514,"updated_at":"2026-10-29 21:57:43.459028-06","commit_sha":"5af1460f4a2dce4f590f6fdfcd108119285e3e7c","content_hash":"67eb1810e7eea071e85e8edab747fd98247dd872","votes":0,"credit":[{"name":"alanoo81 (recipe, graft, measurements)","url":"https://github.com/alanoo81/deepseek-v41-flash-4x-cmp170hx"},{"name":"GNmac-lab (base recipe 1.2.0 @ 2ad5741ab1, MIT)","url":"https://github.com/GNmac-lab/deepseek-v41-flash-1017-cmp170hx"},{"name":"LibertAIDAI (EXL3 2.9bpw @ 0be63c33 / 3.5bpw @ 04a492fc)","url":"https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw"},{"name":"Curtis White (hybrid layout + quantization_config.json)","url":"https://github.com/CurtisAccelerate/DeepSeekv41-Flash-1017-EXL3-CMP170HX"},{"name":"vcruz305/vllm-exl3 (AGPL-3.0-only, inside the built image)","url":"https://github.com/vcruz305/vllm-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81.json","alternate":"/r/deepseek-v4-1-flash-1017-fidelity-first-4x-cmp-170hx-alanoo81"}},{"slug":"qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f","title":"Qwen3.8-Flash-Next W4A16-Merlin on 4x RTX 3090 \u2014 TP2xPP2+EP, MTP3, FP8 KV, 925K pool (halt95)","summary":"Four-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in halt95's W4A16-Merlin build (Intel AutoRound INT4 + FP8 PLE + MTP draft, @ 17284bf): TP2 x PP2 with expert parallel, manual layer partition 25,23, FP8 KV with a 924,993-token pool (three full 262K sessions resident; block-size 4096, prefix-match-unit 64), MTP 3-token probabilistic draft with RecoverSSM, host-mapped PLE pull transport, host-resident embeddings, side-cache layout + Model Runner V2, 42 images/request. Agent-focused: 92.5% of follow-up prompt tokens from cache across 8 sessions, median follow-up first token 1.61 s (0.96 s solo). Maintainer-measured (submitter T0, v2.5.1 bench card 2026-10-06): 5,219-5,533 tok/s cold prefill 10K-261K, 158-174 tok/s single-stream decode (thinking on), 534 tok/s aggregate at 8 concurrent, 262K needle exact, GSM8K 198/200. Engine is a SOURCE BUILD: vLLM v0.30.0 + 97 commits (release bundle verified by commit AND tree hash); argv verbatim from scripts/serve-v2.5.sh @ 28cbb72.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":168.099999999999994315658113919198513031005859375,"prefill_tok_s":5360,"updated_at":"2026-10-29 21:32:28.596454-06","commit_sha":"d2343f0bc9d171b68356492709d3915f2e0eb610","content_hash":"8aefc67a1e986e515743a30617a02bbc169edc71","votes":0,"credit":[{"name":"halt95","url":"https://github.com/halt95/merlin-engine"},{"name":"Qwen (Qwen3.8-Flash-Next)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"Intel AutoRound (INT4 experts)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"RadixArk (FP8 PLE table)","url":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4"},{"name":"DominikBucko (MTP INT4 packing recipe)","url":"https://github.com/DominikBucko/qwen38-flash-next-2x3090"},{"name":"peakcrosser7 (Flash-Next vLLM branch)","url":"https://github.com/peakcrosser7/vllm/commits/release/qwen38next_offload"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f.json","alternate":"/r/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f"}},{"slug":"qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g","title":"Qwen3.8-Flash-Next W4A16-AutoRound DGX Spark vLLM MTP4 Iter7g","summary":"Recipe by kyaky. Qwen3.8-Flash-Next W4A16-AutoRound hybrid (INT4 experts + FP8 side layers + INT8 head; 51B-token n-gram PLE table on NVMe mmap) on one DGX Spark (GB10, 121 GB unified memory) on a verified-deterministic vLLM floor (image qwen38-flash-dgx:iter7g-20261015 on the digest-pinned vllm/vllm-openai:qwen38-flash-next base): MoE align arrival-order shim, drafter-Marlin atomicAdd fix, pinned triton config, compile-cache mount. Measured line v19a-w5 = MTP=4 window-5 block verification: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 streams (+9.2% N1 / +4.8% N8 over MTP=3, two boots per arm, same image). Promoted serving line v20-fp8: fp8 KV cache on a 24 GB pool at native 262,144 context, 1,291,120-token capacity. Chinese +31.3% via a 25,862-id (39.5%) Han draft-vocab rebalance (prose-zhen 34.8 to 45.7 tok/s). Stochastic-correctness gate 37/37 vs bit-exact fp64 oracle; per-round raw data + scripts ship in-repo; copy-agentic-1 failure disclosed.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":61.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:28.107205-06","commit_sha":"dc8982acabf5e6b6417f4871ad5149d61f01755d","content_hash":"e6a2cfdf4903c7ca1b80df3d0f496d3ca3f13840","votes":0,"credit":[{"name":"kyaky","url":"https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g.json","alternate":"/r/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g"}},{"slug":"qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x","title":"Qwen3.8-Flash-Next FP4-Local 1x DGX Spark vLLM b12x","summary":"Recipe by sojufx. Serves NVIDIA's nvidia/Qwen3.8-Flash-Next-FP4-Local (96 GiB NVFP4 quantization-aware-distilled checkpoint in a new compressed NVFP4 scale format) on one DGX Spark (GB10) via the digest-pinned eugr/spark-vllm-b12x community container (reported vLLM 0.1.dev21830+g8244d1c44.d20261015) - the model card requires community b12x containers until the format lands upstream. FP8 KV cache, exact MTP K=3 with standard rejection sampling, 262,144-token context, 4 scheduler slots, 4096-token batched prefill, disk-backed io_uring PLE, prefix cache with Mamba align, vision plus qwen3_xml tool calling, thinking off by default, localhost-only 127.0.0.1:8001 OpenAI-compatible backend with a deliberate no-replace/no-port-take guard. Measured 2026-10-15/16 on the author's Spark: 66.2 tok/s single-stream sparkDash counting decode, 166.4 tok/s at C4, 41-54 tok/s single-stream on the four-prompt wall-clock suite - different measurement bases, not interchangeable.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":66.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:27.177524-06","commit_sha":"5fc0d101d8e9b1c6a1419f3d465e21649f4ecbad","content_hash":"be42677d0cc5ca7f71fe72210e756f1622fe3604","votes":0,"credit":[{"name":"sojufx","url":"https://github.com/sojufx/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark-Recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x.json","alternate":"/r/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x"}}]}