{"meta":{"count":7,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["rtx-3090-x2","170hx"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T14:38:49+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm","title":"Qwen3.8-Flash-Next 3bpw FP8-PLE 1x CMP 170HX DFlash2 RecoverSSM","summary":"Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s editing code, 799.5/791.3/586.4 across eight, up to 48 requests at once - 1,384 tok/s total with the default DFlash2 drafter (1,403 with the checkpoint's MTP head), 4,243 tok/s cold prefill, 292,784-token KV pool. Serves klee100's 3 bpw AutoRound checkpoint (3-bit g128 / 2-bit g64 experts, 47 GiB on card) with halt95's FP8 n-gram table in HOST RAM (48 GiB, hard-linked at build) on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) so the whole decode step is one CUDA graph on a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3 by request count) + the recipe's RecoverSSM + BF16 GDN-state patches (9 files mounted over the image) fit 48 concurrent requests on one card. Measured 2026-10-23 under the MiaAI-Lab protocol.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":197.80000000000001136868377216160297393798828125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.955646-06","commit_sha":"af7d7dde12376551476c8b9e36cd72c68dc32408","content_hash":"c82e80ffa9955d905b61bda381cf7d89db1350ba","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/Mia-AiLab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json","alternate":"/r/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm"}},{"slug":"deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram","title":"DeepSeek-V4.1-Flash-1017 dsx 2.6-bit GPTQ 2x RTX 3090 (Engram)","summary":"DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) quantized to 187.6 GB single GGUF (2.6127 bpw routed-expert average: 5717 PTQ1_0 / 2545 Q2_0 / 7098 Q3_K per-expert GPTQ with allocator-spent bit budget; dense Q8_0 attention/shared expert/head, F16 embedding/compressor/indexer/Engram-wkv, F32 norms/router/mHC), served by dsx, a Rust+CUDA engine built for this checkpoint, on two RTX 3090s with 192 GB RAM. ~26 tok/s decode / ~170 tok/s prefill at --ctx 262144 with all 15,360 routed experts resident. Exact routing: no expert pruning, no router bias - the rank file orders placement only. Engram n-gram tables (~203 GB) are NOT quantized: read at runtime from the official safetensor shards by a prefetch thread. Hottest ~3,300 experts in VRAM across both GPUs (GPU0's set adapts to the conversation, DSX_ADAPT=8 swaps/token), remaining experts computed on 14 CPU threads from a hugepage RAM arena. Quality vs FP8/FP4 teacher (dsx eval, Lucebox 1017 set): KL 0.1972, top-1 82.59%, PPL 9.3626.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":26,"prefill_tok_s":170,"updated_at":"2026-10-28 15:52:50.29183-06","commit_sha":"942f2d4b4ca3b3ca6697019daf09d9615a47b0af","content_hash":"57641e58a944d825b6d88744c42b9451d8ee916a","votes":0,"credit":[{"name":"lxe (dsx engine + 2bit reference quant)","url":"https://github.com/lxe/dsx"},{"name":"icecubetr (2.6-bit GPTQ quant + measurements)","url":"https://huggingface.co/icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit"},{"name":"deepseek-ai (DeepSeek-V4.1-Flash-1017 checkpoint)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"Lucebox (1017 eval set)","url":"https://huggingface.co/Lucebox/DeepSeek-V4.1-Flash-1017-ROCMFP-GGUF"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram.json","alternate":"/r/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram"}},{"slug":"qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4","title":"Qwen3.8-27B AWQ-INT4 + DSpark on RTX 3090s \u2014 SGLang TP1/TP2/TP4","summary":"Validated SGLang serving for the dense Qwen3.8-27B hybrid GDN vision-language model (48 Gated-DeltaNet + 16 gated-attention layers) on RTX 3090 24 GB: one digest-pinned container whose TP env selects tuned 1x/2x/4x presets. AWQ-INT4 W4A16 (Marlin) beats FP8 on Ampere; DSpark speculative decoding (RadixArk 1.36B draft) on 2x/4x with auto-fallback on TP1. Image input everywhere, video on 2x/4x. Author-measured: 136.8 tok/s decode at 8k on 4x (169.8 aggregate at C8), 112.9 on 2x, 43.2 on 1x; up to 128k context on 4x.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":127.099999999999994315658113919198513031005859375,"prefill_tok_s":1372.700000000000045474735088646411895751953125,"updated_at":"2026-10-20 21:52:36.534482-06","commit_sha":"26a1363966d9065a310e9ec7caf124206a25f3f2","content_hash":"3fa45739100d87065553d5fb598424a66d4742fb","votes":0,"credit":[{"name":"sybil-solutions / 0xSero (SGLang presets, image build, benchmarks)","url":"https://github.com/sybil-solutions/qwen-serve"},{"name":"cyankiwi (AWQ-INT4 W4A16 checkpoint)","url":"https://huggingface.co/cyankiwi/Qwen3.8-27B-AWQ-INT4"},{"name":"RadixArk (DSpark speculative drafter)","url":"https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4.json","alternate":"/r/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x RTX 3090 \u2014 vLLM + ExLlamaV3 CPU expert offload, 256K (DominikBucko)","summary":"GLM-5.3-Flash (turboderp EXL3 3.05bpw) tensor-parallel on 2x RTX 3090 24 GB + 128 GB system RAM: vendor vLLM image with the spele1100/vllm-glm53 GLM-5.3 overlay, ExLlamaV3 @ 499890c with EXL3 MoE decode kernels, and CPU expert-offload pools (UVA exact host registers, per-layer LRU expert cache with the hottest 20% pinned, glm53_host.py pool planner, checkpoint lock + k-pool index installer). Bounded 4,096-token thinking budget + preserved reasoning for agent loops, 262,144-token context, prefix caching, OpenAI-compatible API. Docker image + systemd example, reproducible from repro.lock.json pins; measured up to 1,155 tok/s prefill and 47-50 tok/s decode with needles-at-32K/131K/260K + coherence gates passing.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":49.39999999999999857891452847979962825775146484375,"prefill_tok_s":1130.5,"updated_at":"2026-10-20 21:52:36.396387-06","commit_sha":"c93c82b83e6832d41dd10a7037c607a3f4b53d1b","content_hash":"c807eb1b13116b97fedcab43cef2aad9dca165e7","votes":0,"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/glm53-flash-2x3090"},{"name":"turboderp-org (EXL3 quant + ExLlamaV3)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"spele1100 (GLM-5.3 vLLM overlay)","url":"https://github.com/spele1100/vllm-glm53"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp"}},{"slug":"qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix","title":"Qwen3.8-27B INT8 MTP3 prefix-cache on 1x CMP 170HX Hynix","summary":"Qwen3.8-27B at 65-67 tok/s decode (74 structured) with MTP k=3 speculative decoding AND prefix caching simultaneously on a single NVIDIA CMP 170HX 64GB (Hynix HBM2e, GA100/sm80), 262,144-token context on fp8_e4m3 KV, 24/7 production on pinned upstream vLLM nightly 0.26.1rc1.dev1024+g1fe3a1571 \u2014 the first build where prefix caching and MTP spec decode coexist (GDN mamba-align). Cached TTFT on a 7K shared prefix drops ~4.0 s -> 1.44 s (33.5% hit rate) while decode gains 7-9% over the 0.27.1 fork fallback \u2014 the 'MTP vs prefix cache, pick one' tradeoff is gone. Load-bearing trio --async-scheduling + --max-num-batched-tokens 8192 + --max-num-scheduled-tokens 8192 (default chunked-prefill batch = 0.0% hit rate while looking healthy); --prefix-match-unit 400 halves cached TTFT further (must divide the 1600-token GDN block); batch budget 32000 OOMs and hard-wedges the card. Samsung-40GB side recipe (int8 + DFlash2, 97.8 tok/s) with an sm80 int8-repack CPU-fallback patch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":68.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-20 15:34:22.585629-06","commit_sha":"ad3f55dab02cedfb2b187f7dcfc3f1647e0aaa0c","content_hash":"68f95048b88c132d1ddbffc8ceceaf5c38be5140","votes":0,"credit":[{"name":"ahnguyen17 (recipe, patches, probes, A/B benchmarks)","url":"https://github.com/ahnguyen17/cmp-170hx-vllm"},{"name":"lued (Qwen3.8-27B-INT8-W8A16-MTP checkpoint)","url":"https://huggingface.co/lued/Qwen3.8-27B-INT8-W8A16-MTP"},{"name":"vllm-project (vLLM upstream; #50021 GDN spec bounds, #45238 align-mode cache retention)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json","alternate":"/r/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix"}},{"slug":"qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa","title":"Qwen3.8-Flash-Next W4A16-FP8PLE on 2x RTX 3090 \u2014 TP2, expert-offload, MTP3 (DominikBucko)","summary":"Two-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in the albucino W4A16-FP8PLE build (Intel AutoRound INT4 backbone + FP8 PLE table + INT4 MTP draft, checkpoint @ ef55414): TP2 + expert parallel on the humming MoE path, UVA CPU offload (30 GB of cold experts in RAM), dynamic LRU expert cache (hot-84), BF16 KV at full 262,144-token context, MTP 3-token speculative draft, prefix caching. Author-measured (submitter T0, 128 GB RAM host): 131K prompt first token in 31.3 s and up to 4,191 tok/s prefill on the 8K-chunk prefill profile; agent 128K profile decodes 104.8-111.5 tok/s; fast 256K profile reads 260,096 tokens in 90.8 s (2,865 tok/s). Also ships 64 GB RAM and opt-in uncensored (abliteration) profiles. Engine is the pinned vLLM runtime image 0.1.dev20073+g8e685d198, republished as a release image pinned by digest; argv transcribed verbatim from scripts/serve-container.sh @ 939905b.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":106.5,"prefill_tok_s":2947,"updated_at":"2026-10-20 15:34:08.646082-06","commit_sha":"bb7322bf4f7bdf564d617b1ab9f9d2985bf71877","content_hash":"5e0fd674c07293431c87af75dcd971af50a99d37","votes":0,"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/qwen38-flash-next-2x3090"},{"name":"albucino (checkpoint)","url":"https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE"},{"name":"Intel AutoRound (W4A16 target)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"RadixArk (FP8 PLE table)","url":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa.json","alternate":"/r/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa"}},{"slug":"qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n","title":"Qwen3.8-Flash-Next 3bpw MTP on 1x CMP 170HX SSD PLE AIO (iIIusi0n)","summary":"First single-CMP-170HX recipe for Qwen3.8-Flash-Next: the AutoRound 3bpw+MTP checkpoint on ONE 64 GiB SM80 GPU with the 95.37 GiB BF16 PLE table left ON the SSD \u2014 native Linux AIO async reads, 16K prompt read-ahead and a 512 MiB row cache stream it per-row while the model loads metadata-only, so the 176B MoE fits one card. Patch adds async PLE reads/read-ahead, metadata-only loading, CUDA-graph host breaks, mixed-bit expert handling and MTP layer remapping; KLD 0.1208 vs original weights. Three-round medians (128 output tokens, temp 0, thinking off, EOS ignored): 8K prefill 2,636 tok/s, single-request decode 111 tok/s, 667 tok/s aggregate at 16 concurrent. Honest state: 31 SSD/graph tests PASS, 134 requests no API errors, BUT an arithmetic-consistency issue remains open; context advertised 262K, timing tested only through 32K. Tested sparse (15 GiB RAM, 31 GiB swap) on ENTERPRISE SATA at 180 W. Documented swap: the orcarouter uncensored AutoRound 3bpw checkpoint runs the same stack.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":111,"prefill_tok_s":2132,"updated_at":"2026-10-08 21:11:56.66944-06","commit_sha":"827bfe01549b43357fa06a8610ea5e64b55583b0","content_hash":"3a31e6a965360ba192fd941458c89f62f07e27e6","votes":0,"credit":[{"name":"iIIusi0n (recipe, vLLM ple-ssd patch set, AIO helper, benchmark harness)","url":"https://github.com/iIIusi0n/qwen3.8-flash-next-cmp170hx"},{"name":"Qwen (official Qwen3.8-Flash-Next base, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"klee100 (AutoRound 3bpw+MTP quantized checkpoints, official + uncensored)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"OrcaRouter (Qwen3.8-Flash-Next-Uncensored BF16 abliterated fine-tune)","url":"https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored"},{"name":"vLLM project (upstream vLLM @ a5a30471 the patch applies to)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n.json","alternate":"/r/qwen3-8-flash-next-3bpw-mtp-on-1x-cmp-170hx-ssd-ple-aio-iiiusi0n"}}]}