{"meta":{"count":7,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["dgx-spark-x4","rtx-3090-x2"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T18:26:31+00:00"},"recipes":[{"slug":"glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4","title":"GLM-5.3-Flash exlr8 EXL3 experts on DGX Spark TP1-4","summary":"GLM-5.3-Flash exlr8: MoE layers are EXL3-trellis units decoded by the custom exlr8 MoE kernel; dense linears stay FP8/NVFP4. One release carries three expert widths (k2/k3/k4 = 2/3/4 bpw), chosen at boot via VLLM_EXLR8_WIDTHS. Reference topology TP=4 on four GB10 boxes; TP=1 serves the whole model (~87 GiB) on one 128 GB DGX Spark / ASUS GX10 via tp1.yaml plus a distinct MENTAT_GROUP. TP=2 and TP=3 also measured. 2026-10-19, clocks TP-locked at 2200 MHz: decode code 128.0 tok/s, prose 72.5 at TP=4; 53.0 / 29.5 at TP=1; prefill 4,393 tok/s at 8K, 4,190 at 32K. Trade vs the shipping NVFP4 build: decode +3-8%, NLL equal or better, KV pool ~+15%, prefill 7.7% lower at Second 1x-Spark GLM-5.3-Flash lane: not the VQ Q4_K GGUF lane (/r/glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn), and not a fork of the 2x ATLAS build (/r/glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m) whose NVFP4 Flash base this stack overlays.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":65.2000000000000028421709430404007434844970703125,"prefill_tok_s":3438.5,"updated_at":"2026-10-31 11:51:31.886893-06","commit_sha":"e576a851c441c37ac2fb42dfa0188eef41f910e0","content_hash":"237f8f6570a37cb5ddb895edbae6d611ef23660e","votes":0,"credit":[{"name":"kindlingai","url":"https://github.com/kindlingai/glm-5.3-flash-exlr8-gx10"},{"name":"nvidia","url":"https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4.json","alternate":"/r/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4"}},{"slug":"deepseek-v4-1-flash-on-4x-dgx-spark-tuned-tp4-sglang-dspark","title":"DeepSeek-V4.1-Flash on 4x DGX Spark \u2014 tuned TP4 SGLang/DSpark","summary":"","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":89.7000000000000028421709430404007434844970703125,"prefill_tok_s":5650,"updated_at":"2026-10-29 21:32:23.622356-06","commit_sha":"ddf188add74be5de70f13f6f71735ec18a87c83c","content_hash":"647b167cbdd40dfd95cbaabc81e8624e929d847d","votes":0,"credit":[{"name":"knapcio (TP4 profile, v2.1-v2.3 production stack, release gates)","url":"https://github.com/knapcio/DeepSeek-V4.1-Flash-4x-DGX-Spark-TP4"},{"name":"MiaAI-Lab (upstream: launcher, Engram store, b12x routing, memory model)","url":"https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-DGX-Sparks"},{"name":"0xSero (original 4x RTX PRO 6000 recipe skeleton boot.py derives from)","url":"https://github.com/sybil-solutions/deepseek-v4.1-flash-4x-rtx-pro-6000"},{"name":"kpham-sgl (sglang#39187 bounded dense-indexer prefill transient, backported as adapter/indexer_chunked*.py)","url":"https://github.com/sgl-project/sglang/pull/39187"},{"name":"BBuf + dsv4.1 branch contributors (#39370, #39646, #39648, #39653 decode kernel work in the canary images)","url":"https://github.com/sgl-project/sglang"},{"name":"rhys101 (SG17 RoCEnante overlay + SG18 prefill TP split from DeepSeek-V4.1-Flash-vLLM-DGX-Spark-8)","url":"https://github.com/rhys101/DeepSeek-V4.1-Flash-vLLM-DGX-Spark-8"},{"name":"local-inference-lab \u2014 Luke Alonso & Jason (b12x: RoCEnante one-shot RDMA all-reduce + fused MoE kernels)","url":"https://github.com/local-inference-lab/b12x"},{"name":"FujitsuPolycom + sparkring contributors (switchless-ring opposite-node paths, path-aware RoCEnante)","url":"https://github.com/FujitsuPolycom/sparkring"},{"name":"Saolence (#1/#2/#4/#5/#6: ring as opt-in switch, NCCL/build fixes, first-deployer review)","url":"https://github.com/knapcio/DeepSeek-V4.1-Flash-4x-DGX-Spark-TP4/pull/4"},{"name":"rsync/rchmagos (#8: RoCEnante on the switchless ring, DSV41_ROCE_RING + scripts/ring_mesh/)","url":"https://github.com/knapcio/DeepSeek-V4.1-Flash-4x-DGX-Spark-TP4/pull/8"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-on-4x-dgx-spark-tuned-tp4-sglang-dspark.json","alternate":"/r/deepseek-v4-1-flash-on-4x-dgx-spark-tuned-tp4-sglang-dspark"}},{"slug":"deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram","title":"DeepSeek-V4.1-Flash-1017 dsx 2.6-bit GPTQ 2x RTX 3090 (Engram)","summary":"DeepSeek-V4.1-Flash-1017 (552B MoE + 196B Engram) quantized to 187.6 GB single GGUF (2.6127 bpw routed-expert average: 5717 PTQ1_0 / 2545 Q2_0 / 7098 Q3_K per-expert GPTQ with allocator-spent bit budget; dense Q8_0 attention/shared expert/head, F16 embedding/compressor/indexer/Engram-wkv, F32 norms/router/mHC), served by dsx, a Rust+CUDA engine built for this checkpoint, on two RTX 3090s with 192 GB RAM. ~26 tok/s decode / ~170 tok/s prefill at --ctx 262144 with all 15,360 routed experts resident. Exact routing: no expert pruning, no router bias - the rank file orders placement only. Engram n-gram tables (~203 GB) are NOT quantized: read at runtime from the official safetensor shards by a prefetch thread. Hottest ~3,300 experts in VRAM across both GPUs (GPU0's set adapts to the conversation, DSX_ADAPT=8 swaps/token), remaining experts computed on 14 CPU threads from a hugepage RAM arena. Quality vs FP8/FP4 teacher (dsx eval, Lucebox 1017 set): KL 0.1972, top-1 82.59%, PPL 9.3626.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":26,"prefill_tok_s":170,"updated_at":"2026-10-28 15:52:50.29183-06","commit_sha":"942f2d4b4ca3b3ca6697019daf09d9615a47b0af","content_hash":"57641e58a944d825b6d88744c42b9451d8ee916a","votes":0,"credit":[{"name":"lxe (dsx engine + 2bit reference quant)","url":"https://github.com/lxe/dsx"},{"name":"icecubetr (2.6-bit GPTQ quant + measurements)","url":"https://huggingface.co/icecubetr/DeepSeek-V4.1-Flash-1017-dsx-2.6bit"},{"name":"deepseek-ai (DeepSeek-V4.1-Flash-1017 checkpoint)","url":"https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017"},{"name":"Lucebox (1017 eval set)","url":"https://huggingface.co/Lucebox/DeepSeek-V4.1-Flash-1017-ROCMFP-GGUF"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram.json","alternate":"/r/deepseek-v4-1-flash-1017-dsx-2-6-bit-gptq-2x-rtx-3090-engram"}},{"slug":"qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4","title":"Qwen3.8-27B AWQ-INT4 + DSpark on RTX 3090s \u2014 SGLang TP1/TP2/TP4","summary":"Validated SGLang serving for the dense Qwen3.8-27B hybrid GDN vision-language model (48 Gated-DeltaNet + 16 gated-attention layers) on RTX 3090 24 GB: one digest-pinned container whose TP env selects tuned 1x/2x/4x presets. AWQ-INT4 W4A16 (Marlin) beats FP8 on Ampere; DSpark speculative decoding (RadixArk 1.36B draft) on 2x/4x with auto-fallback on TP1. Image input everywhere, video on 2x/4x. Author-measured: 136.8 tok/s decode at 8k on 4x (169.8 aggregate at C8), 112.9 on 2x, 43.2 on 1x; up to 128k context on 4x.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":127.099999999999994315658113919198513031005859375,"prefill_tok_s":1372.700000000000045474735088646411895751953125,"updated_at":"2026-10-20 21:52:36.534482-06","commit_sha":"26a1363966d9065a310e9ec7caf124206a25f3f2","content_hash":"3fa45739100d87065553d5fb598424a66d4742fb","votes":0,"credit":[{"name":"sybil-solutions / 0xSero (SGLang presets, image build, benchmarks)","url":"https://github.com/sybil-solutions/qwen-serve"},{"name":"cyankiwi (AWQ-INT4 W4A16 checkpoint)","url":"https://huggingface.co/cyankiwi/Qwen3.8-27B-AWQ-INT4"},{"name":"RadixArk (DSpark speculative drafter)","url":"https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4.json","alternate":"/r/qwen3-8-27b-awq-int4-dspark-on-rtx-3090s-sglang-tp1-tp2-tp4"}},{"slug":"glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2","title":"GLM-5.3-Flash NVFP4 4x DGX Spark switchless ring TP4 DFlash2","summary":"Recipe by Alex Ellis / OpenFaaS Ltd. GLM-5.3-Flash (320B-A18B, NVFP4) served tensor-parallel 4 across four DGX Spark (GB10, sm_121) nodes cabled as a closed switchless RoCE ring: two rails per node wired to two neighbours, the non-adjacent hop relayed through a neighbour \u2014 no switch on the data path or the procurement path. Collectives use canonical switchless-nccl v0.0.1 (NCCL 2.30.7 + two Apache-2.0 switchless patches + OpenFaaS hardening patch), LD_PRELOADed with pinned library sha256 and NCCL_ALGO=Ring; not SparkRing/SIRCL, distinct from the site's switched 4x Spark SGLang lane. vLLM on the public tonyd2wild sm121-v11-dflash2 image: TP4, marlin MoE, FP8 E4M3 KV at 12 GiB per rank (pool 1,576,246 tokens = 6.01x window), DFlash2 drafter k=7, glm47 tool + glm45 reasoning parsers, max-num-seqs 6, workers (ranks 3,2,1) headless first, then head rank 0 opens the single OpenAI-compatible endpoint on :8000. Same fabric also served GLM-5.2 (EXL3, QuantTrio) and DeepSeek-V4-Flash at TP4.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":45,"prefill_tok_s":1971.59999999999990905052982270717620849609375,"updated_at":"2026-10-20 21:52:36.421715-06","commit_sha":"3ec203d32cde9b249d3e88f040a8d0c79650c3db","content_hash":"9cf1d9bb8001b5e721793f15e201aee0b5916f6d","votes":0,"credit":[{"name":"Alex Ellis / OpenFaaS Ltd","url":"https://github.com/alexellis/glm-5.3-flash-4x-dgx-spark-switchless"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2.json","alternate":"/r/glm-5-3-flash-nvfp4-4x-dgx-spark-switchless-ring-tp4-dflash2"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp","title":"GLM-5.3-Flash EXL3 3.05bpw on 2x RTX 3090 \u2014 vLLM + ExLlamaV3 CPU expert offload, 256K (DominikBucko)","summary":"GLM-5.3-Flash (turboderp EXL3 3.05bpw) tensor-parallel on 2x RTX 3090 24 GB + 128 GB system RAM: vendor vLLM image with the spele1100/vllm-glm53 GLM-5.3 overlay, ExLlamaV3 @ 499890c with EXL3 MoE decode kernels, and CPU expert-offload pools (UVA exact host registers, per-layer LRU expert cache with the hottest 20% pinned, glm53_host.py pool planner, checkpoint lock + k-pool index installer). Bounded 4,096-token thinking budget + preserved reasoning for agent loops, 262,144-token context, prefix caching, OpenAI-compatible API. Docker image + systemd example, reproducible from repro.lock.json pins; measured up to 1,155 tok/s prefill and 47-50 tok/s decode with needles-at-32K/131K/260K + coherence gates passing.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":49.39999999999999857891452847979962825775146484375,"prefill_tok_s":1130.5,"updated_at":"2026-10-20 21:52:36.396387-06","commit_sha":"c93c82b83e6832d41dd10a7037c607a3f4b53d1b","content_hash":"c807eb1b13116b97fedcab43cef2aad9dca165e7","votes":0,"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/glm53-flash-2x3090"},{"name":"turboderp-org (EXL3 quant + ExLlamaV3)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"spele1100 (GLM-5.3 vLLM overlay)","url":"https://github.com/spele1100/vllm-glm53"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-2x-rtx-3090-vllm-exllamav3-cpu-exp"}},{"slug":"qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa","title":"Qwen3.8-Flash-Next W4A16-FP8PLE on 2x RTX 3090 \u2014 TP2, expert-offload, MTP3 (DominikBucko)","summary":"Two-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in the albucino W4A16-FP8PLE build (Intel AutoRound INT4 backbone + FP8 PLE table + INT4 MTP draft, checkpoint @ ef55414): TP2 + expert parallel on the humming MoE path, UVA CPU offload (30 GB of cold experts in RAM), dynamic LRU expert cache (hot-84), BF16 KV at full 262,144-token context, MTP 3-token speculative draft, prefix caching. Author-measured (submitter T0, 128 GB RAM host): 131K prompt first token in 31.3 s and up to 4,191 tok/s prefill on the 8K-chunk prefill profile; agent 128K profile decodes 104.8-111.5 tok/s; fast 256K profile reads 260,096 tokens in 90.8 s (2,865 tok/s). Also ships 64 GB RAM and opt-in uncensored (abliteration) profiles. Engine is the pinned vLLM runtime image 0.1.dev20073+g8e685d198, republished as a release image pinned by digest; argv transcribed verbatim from scripts/serve-container.sh @ 939905b.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":106.5,"prefill_tok_s":2947,"updated_at":"2026-10-20 15:34:08.646082-06","commit_sha":"bb7322bf4f7bdf564d617b1ab9f9d2985bf71877","content_hash":"5e0fd674c07293431c87af75dcd971af50a99d37","votes":0,"credit":[{"name":"DominikBucko","url":"https://github.com/DominikBucko/qwen38-flash-next-2x3090"},{"name":"albucino (checkpoint)","url":"https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE"},{"name":"Intel AutoRound (W4A16 target)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"RadixArk (FP8 PLE table)","url":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa.json","alternate":"/r/qwen3-8-flash-next-w4a16-fp8ple-on-2x-rtx-3090-tp2-expert-offloa"}}]}