{"meta":{"count":5,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["rtx-3090","170hx-x3"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T14:39:59+00:00"},"recipes":[{"slug":"glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin","title":"GLM-5.3-Flash EXL3 3.05bpw on 1x RTX 3090 \u2014 RAM/NVMe expert tiering (near-exact)","summary":"320B MoE served on ONE 24 GB RTX 3090 with stock exllamav3 1.5.1 plus process-local monkeypatches: the 12,096 routed experts (42 MoE layers x 288, 8 picked per token per layer) tier across VRAM, pinned host RAM and a packed NVMe store. An elastic VRAM cache holds the ~1,300 hottest experts (12.4 GB); evicted experts move into a pinned no-duplicates RAM tier sized from the container cap; a 117 GB O_DIRECT record store covers the rest, with layer-ahead prefetch, a victim write-back ring and an AVX2 CPU lane computing RAM-resident decode picks in place. Modes: fast (all experts in RAM, ~218 GiB free, 28.2 tok/s, KL 0.005), nvme (55 GiB cap, 19.5 tok/s, KL 0.008; 16 GiB cap 14.7 lab / 14.38 sweep), nvme-exact (bit-exact, 8.3 tok/s), and an experimental Intel Arc Pro B70 peer-offload tier (GLM53_B70=1, 26.5 tok/s, KL 0.003 - not lab-run). 131,072-token KV, OpenAI API on port 30000, model id glm-5.3-flash, no output caps. Registry lab six gates pass at 55 GiB (PR #199); 16 GiB reported.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":24.10000000000000142108547152020037174224853515625,"prefill_tok_s":1015,"updated_at":"2026-10-29 21:32:24.549403-06","commit_sha":"53520bbf5e06f65db6670e8693efb018f75026c8","content_hash":"65ab07a0787590e7c21331b006d4a13553080d02","votes":0,"credit":[{"name":"sybil-solutions (image, tiering runtime, omarchy plugin)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"turboderp (ExLlamaV3 + EXL3 3.05bpw checkpoint)","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-3090-ram-nvme-expert-tierin"}},{"slug":"ds4-flash-3x170hx-pp3-dspark-k5","title":"DeepSeek V4 Flash-0731 abliterated on 3x CMP 170HX \u2014 vLLM patched PP3 DSpark K5","summary":"The proven 3-card pipeline-parallel recipe on CMP 170HX (sm_80, 64GB unlocked): PP=3 with manual layer partition 15,16,12 at util 0.98, FP8 KV, block-size 256, DSpark block drafting at exactly 5 speculative tokens, async scheduling, CPU pinning (dedicated physical cores, placeholder in launch args), hard memory cap 64g with swap disallowed. Structured decode 181 tok/s at 99% acceptance. Upstream reference repo: allover326/deepseek-v4-cmp170hx @ 3dd2d88 (Aug 13 2026) \u2014 documents 15,15,13 3-card split, retracted 4-card requirement, INT4 524288 ctx w/ DSV4_LOGITS_ROW_CHUNK=64, +7% decode via c3046d1 rebase patches (full source build required). First-party 15,16,12 numbers on this page are ours.","tier":"curated","beta":false,"generation":"ampere","decode_tok_s":129.900000000000005684341886080801486968994140625,"prefill_tok_s":2350,"updated_at":"2026-10-29 21:32:23.860633-06","commit_sha":"24a9ac90156459e4efb49c951590edbea775df38","content_hash":"02d56ece2249f2b3e2b1ee2ffce158e841e20d26","votes":0,"credit":[{"name":"allover326","url":"https://github.com/allover326/deepseek-v4-cmp170hx"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/ds4-flash-3x170hx-pp3-dspark-k5.json","alternate":"/r/ds4-flash-3x170hx-pp3-dspark-k5"}},{"slug":"qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4","title":"Qwen3.8-27B AWQ-INT4 + DSpark on RTX 3090s \u2014 SGLang TP1/TP2/TP4","summary":"Validated SGLang serving for the dense Qwen3.8-27B hybrid GDN vision-language model (48 Gated-DeltaNet + 16 gated-attention layers) on RTX 3090 24 GB: one digest-pinned container whose TP env selects tuned 1x/2x/4x presets. AWQ-INT4 W4A16 (Marlin) beats FP8 on Ampere; DSpark speculative decoding (RadixArk 1.36B draft) on 2x/4x with auto-fallback on TP1. Image input everywhere, video on 2x/4x. Author-measured: 136.8 tok/s decode at 8k on 4x (169.8 aggregate at C8), 112.9 on 2x, 43.2 on 1x; up to 128k context on 4x.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":127.099999999999994315658113919198513031005859375,"prefill_tok_s":1372.700000000000045474735088646411895751953125,"updated_at":"2026-10-20 21:52:36.534482-06","commit_sha":"26a1363966d9065a310e9ec7caf124206a25f3f2","content_hash":"3fa45739100d87065553d5fb598424a66d4742fb","votes":0,"credit":[{"name":"sybil-solutions / 0xSero (SGLang presets, image build, benchmarks)","url":"https://github.com/sybil-solutions/qwen-serve"},{"name":"cyankiwi (AWQ-INT4 W4A16 checkpoint)","url":"https://huggingface.co/cyankiwi/Qwen3.8-27B-AWQ-INT4"},{"name":"RadixArk (DSpark speculative drafter)","url":"https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4.json","alternate":"/r/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4"}},{"slug":"qwen3-8-flash-next-w4a16-3x-cmp-170hx-pp3-nvme-ple-fp8-kv-1m-cle","title":"Qwen3.8-Flash-Next W4A16 3x CMP 170HX PP3 NVMe PLE fp8 KV 1M (clee)","summary":"Working PP3 recipe for Qwen3.8-Flash-Next (Qwen4Exp 180B-A6B MoE, Intel INT4 W4A16 AutoRound) on 3x CMP 170HX (SM80, unlocked 64 GiB each): the 95.4 GiB BF16 PLE n-gram table streams live from NVMe (shard-direct preadv into a fixed GPU staging buffer, never fully loaded), fp8 e4m3 KV software-decoded on SM80, mamba SSM cache bf16, and 1,048,576-token context via YaRN x4 over the 262144-native checkpoint. KV pool 5.97M tokens (~5.7 concurrent 1M sessions). W4A16 experts run GPTQ-Marlin bit-exact; no GPU P2P on this card pairing, so PP stages through host RAM with NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce. Measured cold-cache temp=0: prefill 2389 tok/s @4k rising to 4690 @1M (QSA sparse crossover past ~64k), decode 73.2 -> 59.8 tok/s @1M, TTFT 223 s @1M; exact needle recall at all depths through 1M. Fork clee/vllm-sm80 on a PINNED nightly; newer nightlies not validated.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":73.2000000000000028421709430404007434844970703125,"prefill_tok_s":3563,"updated_at":"2026-10-20 15:34:25.555323-06","commit_sha":"fb3da33b3299da70eb24ce68a2c772de0dc1a356","content_hash":"0dea5f8b482a14a47e6cef76e85ad322d56f9821","votes":1,"credit":[{"name":"clee (recipe, fork patches, bench + needle harness)","url":"https://github.com/clee/qwen3.8-flash-next-3x170hx"},{"name":"clee (vllm-sm80 fork: units-fix, fp8-kv-sm80, ple-pp3, ple-nvme-staged)","url":"https://github.com/clee/vllm-sm80"},{"name":"Kai / 2Wild (Qwen38-Flash-Next-4x3090 \u2014 staged-PLE architecture inspiration)","url":"https://github.com/tonyd2wild/Qwen38-Flash-Next-4x3090"},{"name":"Intel (INT4 W4A16 AutoRound quant)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"froggeric (bugfixed Qwen3.8 chat templates)","url":"https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-3x-cmp-170hx-pp3-nvme-ple-fp8-kv-1m-cle.json","alternate":"/r/qwen3-8-flash-next-w4a16-3x-cmp-170hx-pp3-nvme-ple-fp8-kv-1m-cle"}},{"slug":"qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts","title":"Qwen3.8-Flash-Next W4A16-FP8PLE on 1x RTX 3090 \u2014 CPU cold experts, MTP3, 128K (DominikBucko)","summary":"Single-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in the albucino W4A16-FP8PLE build (@ ef55414, same checkpoint as the 2x3090 runtime): the GPU holds attention + dense weights + the 32 busiest experts per layer, a pinned AVX2 CPU thread pool computes the other 480 experts straight from RAM during decode (GPU takes ~22% over PCIe), and prefill streams all experts through the GPU in 8,192-token chunks. INT8 KV cache fits the full 135,168-token context in 3.05 GB; MTP 3-token draft; PLE table read in place from NVMe. The most common local-AI machine \u2014 one 24 GB card + 64 GB RAM. Author-measured (submitter T0, release image): 131,099-token prompt reads at 2,106 tok/s (62 s to first token) then decodes 47.4 tok/s; 43-51 tok/s decode range on 32 cores; 32K prompt at 2,141-2,219 tok/s prefill. Same vLLM runtime as the 2x3090 recipe, repackaged for half the hardware, nothing pruned. Argv transcribed verbatim from scripts/serve-container.sh @ 10b24c2.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":48.89999999999999857891452847979962825775146484375,"prefill_tok_s":2123.5,"updated_at":"2026-10-20 15:34:09.15642-06","commit_sha":"088542fd7b452ae34d7f6b0253b939a3ef6b350d","content_hash":"a354418ea2f8a6dbcef09d963e5901758b7dd549","votes":0,"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/qwen38-flash-next-3090"},{"name":"albucino (checkpoint)","url":"https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts.json","alternate":"/r/qwen3-8-flash-next-w4a16-fp8ple-on-1x-rtx-3090-cpu-cold-experts"}}]}