{"meta":{"count":16,"page":1,"pages":2,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["rtx-pro-6000-blackwell","dgx-spark"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T21:09:24+00:00"},"recipes":[{"slug":"glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4","title":"GLM-5.3-Flash exlr8 EXL3 experts on DGX Spark TP1-4","summary":"GLM-5.3-Flash exlr8: MoE layers are EXL3-trellis units decoded by the custom exlr8 MoE kernel; dense linears stay FP8/NVFP4. One release carries three expert widths (k2/k3/k4 = 2/3/4 bpw), chosen at boot via VLLM_EXLR8_WIDTHS. Reference topology TP=4 on four GB10 boxes; TP=1 serves the whole model (~87 GiB) on one 128 GB DGX Spark / ASUS GX10 via tp1.yaml plus a distinct MENTAT_GROUP. TP=2 and TP=3 also measured. 2026-10-19, clocks TP-locked at 2200 MHz: decode code 128.0 tok/s, prose 72.5 at TP=4; 53.0 / 29.5 at TP=1; prefill 4,393 tok/s at 8K, 4,190 at 32K. Trade vs the shipping NVFP4 build: decode +3-8%, NLL equal or better, KV pool ~+15%, prefill 7.7% lower at Second 1x-Spark GLM-5.3-Flash lane: not the VQ Q4_K GGUF lane (/r/glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn), and not a fork of the 2x ATLAS build (/r/glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m) whose NVFP4 Flash base this stack overlays.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":65.2000000000000028421709430404007434844970703125,"prefill_tok_s":3438.5,"updated_at":"2026-10-31 11:51:31.886893-06","commit_sha":"e576a851c441c37ac2fb42dfa0188eef41f910e0","content_hash":"237f8f6570a37cb5ddb895edbae6d611ef23660e","votes":0,"credit":[{"name":"kindlingai","url":"https://github.com/kindlingai/glm-5.3-flash-exlr8-gx10"},{"name":"nvidia","url":"https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4.json","alternate":"/r/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4"}},{"slug":"qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3","title":"Qwen3.8-Flash-Next NVFP4 on RTX PRO 6000 Blackwell \u2014 TP1 MTP3","summary":"Single-card Blackwell recipe for Qwen3.8-Flash-Next (hybrid GDN/QSA MoE, 48 layers, 512 experts) on one 96 GB RTX PRO 6000: the Community image community image R34 (localinferencelab/vllm, v0.26.1rc0+glm53.r34), TP1 with MTP3 speculative decoding (b12x MoE backend), 256k context. Benchmark-qualified on RTX PRO 6000 Blackwell Max-Q: decode 148 tok/s at ctx-0 (temp 1), prefill 3860 tok/s at 2048 batched tokens. Launch argv transcribed verbatim from the r35 community deployment. Weights: JMNI-Labs/Qwen3.8-Flash-Next-NVFP4-QAD5500-Hybrid \u2014 step-5500 QAD trained tensors recombined with main 7c4f1bc1 MXFP8 attention + NVFP4 MTP experts; same-stack measured +5% MTP-normalized decode, +25% with MTP vs the main checkpoint (evals within noise, MMLU-Pro 66.9 vs 67.2).","tier":"curated","beta":false,"generation":"blackwell","decode_tok_s":137.19999999999998863131622783839702606201171875,"prefill_tok_s":6627.3000000000001818989403545856475830078125,"updated_at":"2026-10-29 23:29:38.177885-06","commit_sha":"28709ecba026aff18371426bc26025e5a4185209","content_hash":"1e37b0f978392ba766c01cfdf9ae9ee3e002ee84","votes":1,"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/rtx6kpro"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json","alternate":"/r/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3"}},{"slug":"qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g","title":"Qwen3.8-Flash-Next W4A16-AutoRound DGX Spark vLLM MTP4 Iter7g","summary":"Recipe by kyaky. Qwen3.8-Flash-Next W4A16-AutoRound hybrid (INT4 experts + FP8 side layers + INT8 head; 51B-token n-gram PLE table on NVMe mmap) on one DGX Spark (GB10, 121 GB unified memory) on a verified-deterministic vLLM floor (image qwen38-flash-dgx:iter7g-20261015 on the digest-pinned vllm/vllm-openai:qwen38-flash-next base): MoE align arrival-order shim, drafter-Marlin atomicAdd fix, pinned triton config, compile-cache mount. Measured line v19a-w5 = MTP=4 window-5 block verification: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 streams (+9.2% N1 / +4.8% N8 over MTP=3, two boots per arm, same image). Promoted serving line v20-fp8: fp8 KV cache on a 24 GB pool at native 262,144 context, 1,291,120-token capacity. Chinese +31.3% via a 25,862-id (39.5%) Han draft-vocab rebalance (prose-zhen 34.8 to 45.7 tok/s). Stochastic-correctness gate 37/37 vs bit-exact fp64 oracle; per-round raw data + scripts ship in-repo; copy-agentic-1 failure disclosed.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":61.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:28.107205-06","commit_sha":"dc8982acabf5e6b6417f4871ad5149d61f01755d","content_hash":"e6a2cfdf4903c7ca1b80df3d0f496d3ca3f13840","votes":0,"credit":[{"name":"kyaky","url":"https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g.json","alternate":"/r/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g"}},{"slug":"qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp","title":"Qwen3.8-Flash-Next NVFP4 DGX Spark GB10 SGLang NEXTN MTP","summary":"Production single-node DGX Spark (Grace Blackwell GB10, SM121, 128GB unified memory) recipe for Qwen3.8-Flash-Next on SGLang: ModelOpt NVFP4 quantization with zero-copy NVMe mmap of the 47.7 GiB ple_table.bin via --ple-offload-embedding (eliminates 51 GiB of unified-memory footprint), custom Triton FP4 prefill kernels + trtllm_mha decode kernels on SM121, NEXTN MTP (3 verification steps / 4 draft tokens) lifting output to 110-152 tok/s, GDN+QSA hybrid (131,072-token served window, 262k native capacity), 96-slot mamba recurrence buffers, sharp froggeric v22.1 Jinja chat template (-18% response length), persistent Triton/Inductor SM121 JIT caches on NVMe, 8-worker parallel NVMe pre-warm (206 shards in 17s) + 16-core Grace OMP. Author-measured vs same-box 27B dense baseline: 110.4-152.8 tok/s decode w/ MTP (baseline 12.1 base / 21.5 MTP), single-stream CoT 25.1-28.8 tok/s (2.62x speculative gain), TTFT ~347ms radix hit / ~465ms cold, 113 GiB / 121 GiB RAM used (~8 GiB headroom), 103.5 GiB","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":110.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:27.869274-06","commit_sha":"2c5220aa85b5379d7373abd0aa14bb13b1ec993f","content_hash":"bf0c02f95b99494d6720a515e2f8f438f126bc87","votes":0,"credit":[{"name":"Michael Bemler (Bemler Labs)","url":"https://github.com/bemlerlabs/qwen3.8-flash-next-dgx-spark-sglang"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp.json","alternate":"/r/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp"}},{"slug":"qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x","title":"Qwen3.8-Flash-Next FP4-Local 1x DGX Spark vLLM b12x","summary":"Recipe by sojufx. Serves NVIDIA's nvidia/Qwen3.8-Flash-Next-FP4-Local (96 GiB NVFP4 quantization-aware-distilled checkpoint in a new compressed NVFP4 scale format) on one DGX Spark (GB10) via the digest-pinned eugr/spark-vllm-b12x community container (reported vLLM 0.1.dev21830+g8244d1c44.d20261015) - the model card requires community b12x containers until the format lands upstream. FP8 KV cache, exact MTP K=3 with standard rejection sampling, 262,144-token context, 4 scheduler slots, 4096-token batched prefill, disk-backed io_uring PLE, prefix cache with Mamba align, vision plus qwen3_xml tool calling, thinking off by default, localhost-only 127.0.0.1:8001 OpenAI-compatible backend with a deliberate no-replace/no-port-take guard. Measured 2026-10-15/16 on the author's Spark: 66.2 tok/s single-stream sparkDash counting decode, 166.4 tok/s at C4, 41-54 tok/s single-stream on the four-prompt wall-clock suite - different measurement bases, not interchangeable.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":66.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:27.177524-06","commit_sha":"5fc0d101d8e9b1c6a1419f3d465e21649f4ecbad","content_hash":"be42677d0cc5ca7f71fe72210e756f1622fe3604","votes":0,"credit":[{"name":"sojufx","url":"https://github.com/sojufx/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark-Recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x.json","alternate":"/r/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x"}},{"slug":"nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k","title":"Nemotron 3.5 Bolt 74B-A9B 1x DGX Spark vLLM b12x LiLiCorr 262K","summary":"NVIDIA Nemotron 3.5 Bolt 74B-A9B-24GB (hybrid Mamba-2/Transformer latent-MoE, 74B with 9B active, IQ2_XXS 2-bit experts, NVFP4 Mamba layers, Q8_0 rest; text+image+video in, 262,144-token window, OpenMDW-1.1) on one DGX Spark (GB10/SM121) with vLLM nightly-77154ca6 (base image vllm/vllm-openai@sha256:1ee85ca8) + b12x 1.5.1 kernels (--linear-backend b12x --moe-backend b12x), FP8 KV cache, gpu-memory-utilization 0.70, max-num-seqs 16, and the NVIDIA LiLiCorr MTP drafter at 7 speculative tokens. Pinned revisions: model c6cc2e8f, drafter 4c6a5613. Measured on the reference box: ~140 tok/s code rewrite / 63 summarise / 50 prose single-stream with the drafter (mean acceptance 4.0 tokens/pass), 31 tok/s without; 74.8/113.4/163.6/201.2 tok/s at 1/4/8/16 streams; KV cache ~6.52M tokens at the 0.70 share. On a 64 GB Spark the lane takes its own memory share 0.60: 667,874 KV tokens (2.55 windows), 110.8 tok/s greedy, 7.1 GiB left at the peak (2026-10-24).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":63,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.717952-06","commit_sha":"2cdb5f11ffa1fb1af8676496fdd631976c3d536d","content_hash":"28887fb2cf3bab1914c803e792898fff92b9afe0","votes":0,"credit":[{"name":"hasso5703","url":"https://github.com/hasso5703"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k.json","alternate":"/r/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k"}},{"slug":"qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic","title":"Qwen3.8-27B SGLang NVFP4 DFlash2 on 1x DGX Spark (deterministic)","summary":"First dense Qwen3.8-27B recipe on 1x DGX Spark (GB10, 128 GB unified): SGLang v0.5.22 release image + NVFP4 target + DFlash2 speculative decoding (NVFP4 calibrated 5-layer draft, depth 16), served by a boot-persistent systemd unit from a one-command installer. Author-measured 68.3 tok/s greedy median single stream (bench.sh, thinking on; v1.39.0 A/B 2026-10-23), 81.7 tok/s at 4 streams (serving-bench), 135-148 tok/s at 8 streams and 258 at 32. Fully pinned (image digest, checkpoint revisions, draft depth) and reproducible to the decimal across boots via --disable-flashinfer-autotune (\"The boot lottery\", BENCHMARKS.md); 1M-token YaRN context by default, native 262144 as fallback; OpenAI- and Anthropic-compatible API on :30001 through the box's keepalive proxy (the engine itself answers on 127.0.0.1:30000, on the box only), with opencode wired out of the box.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":68.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-10-28 16:49:58.824279-06","commit_sha":"307fa2334d111e38258e27fa9e690c44e96b441b","content_hash":"7d8b3e291e1b8abfa9db4ded3e90cd601b8bba78","votes":0,"credit":[{"name":"hasso5703","url":"https://github.com/hasso5703"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic.json","alternate":"/r/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic"}},{"slug":"qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode","title":"Qwen3.8-27B NVFP4A4-AWQ 1x RTX PRO 6000 SGLang Fast Decode","summary":"Recipe by AppNetDeveloper. ThinkingCap-Qwen3.8-27B-NVFP4A4-AWQ on 1x RTX PRO 6000 Blackwell (SM120) with stock SGLang 0.5.22 plus engine overlays, no fork: FR-Spec pruned draft lm_head for a quantized FP8 head (slices weight AND per-row scale along vocab, bit-identical kept rows); KDA NVFP4 SM120 GEMM allowlist extension (KN=5120x34816,17408x5120) 85-89% DRAM bandwidth at M<=16 vs 50-70% default; tuned triton_scaled_mm fp8_w8a8_channelwise JSONs. On 0.5.22 the overlays land as ONE patch, patches/sglang-0.5.22/frspec_kda_sglang-0.5.22.patch (patch -p1). NEXTN/MTP, FP8_e4m3 KV cache, HiCache, 40k hot-token draft list; hybrid GDN+MTP needs --enable-linear-replayssm-spec. Author bench 2026-10-14 (400 output tokens, unique prefixes): 131.7 tok/s 1req 45k ctx vs 98.5 (+34%); 4req@45k +13.5%; 8req@27k +3%; 1req@141k +24%; same quality, no extra memory. MTP5 + FR-Spec is the author production default; measured on real agent text: 149 vs 140 @1req, 117 vs 103 @2req, 77 vs 69 @4req over MTP4.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":131.69999999999998863131622783839702606201171875,"prefill_tok_s":0,"updated_at":"2026-10-28 16:25:30.123672-06","commit_sha":"607c8d8babf1cd4237a56128e0614aceb2176332","content_hash":"ba21205503ad8f52405ab4becff180d4e348b45f","votes":0,"credit":[{"name":"AppNetDeveloper","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode"},{"name":"SGLang 0.5.22 port + MTP5/FR-Spec real-text results @ c00cfec525 (published upstream 2026-10-21)","url":"https://github.com/AppNetDeveloper/sglang-sm120-fast-decode/commit/c00cfec525efb20e9c4185ca6c86e61a3235dbc1"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode.json","alternate":"/r/qwen3-8-27b-nvfp4a4-awq-1x-rtx-pro-6000-sglang-fast-decode"}},{"slug":"qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data","title":"Qwen3.8-Flash-Next NVFP4 1x DGX Spark vLLM 71.3 c1 (salient-data)","summary":"Qwen3.8-Flash-Next (~180B multimodal MoE, NVFP4) on one NVIDIA DGX Spark (GB10, 121 GiB unified memory) with a pinned vLLM nightly image + 14 provenance-tracked patches (patch --fuzz=0) + one docker run, fitting the 51.2 GB PLE n-gram table by never making it resident (rows read per forward-pass step from the checkpoint file). 69.3 tok/s single-stream median decode (runs 66.8/69.5/69.3), 41.0 prose median, 91.9 tok/s 6-stream aggregate (1 run), GSM8K 99/100 on a fixed 100-problem subset (temp 0.6 top-p 0.95 thinking on), 12/12 arithmetic + 131K needle pass + prefix-cache canary pass, prefix caching ON. Measured with tonyd2wild's bench_categories.py @ 6ad1c8f unchanged on build 2026-10-22-8e2beaa3; clean first-time run 2026-10-25 re-measured 69.8/41.5/82.6. Prod/dev: prod image digest-pinned + cosign-signed + SPDX SBOM; dev rebuilds the same serving code from published patches with an AST-equivalence proof. Outside the model card's support statement (B200/B300 listed, GB10 unlisted).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":69.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-10-28 15:59:39.152527-06","commit_sha":"85e2e4120ef69829bff82736c39745ac579c9a6b","content_hash":"63f86642d07e8833929005e2087044adaecc1ca4","votes":0,"credit":[{"name":"Jeremy Newhouse / salient-data (recipe, patches, measurements)","url":"https://github.com/salient-data/qwen38-next-dgx-recipe"},{"name":"tonyd2wild (bench_categories.py @ 6ad1c8f + comparator stack/numbers)","url":"https://github.com/tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark"},{"name":"NVIDIA (Qwen3.8-Flash-Next-NVFP4 checkpoint; HF token gated)","url":"https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4"},{"name":"vLLM project (nightly base image 8e2beaa3)","url":"https://github.com/vllm-project/vllm"},{"name":"Sigstore/cosign (image signing + SBOM timestamping)","url":"https://docs.sigstore.dev"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data.json","alternate":"/r/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data"}},{"slug":"qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang","title":"Qwen3.8-27B Bit-Exact Spec-Decode 1x RTX PRO 6000 SGLang","summary":"Recipe by bill-h-lin. Qwen3.8-27B BF16 (BF16 KV, temperature 0) on one RTX PRO 6000 Blackwell Max-Q (SM120) with SGLang made bit-exact under speculative decoding: verify GDN recurrence routed through the decode kernel (old verify kernel tiled state differently, kept beta fp32), verify attention run as decode rows, a DeepGEMM sm120 bf16 small-M GEMM fence for repeatability (upstream from 0.2.3), and upstream sgl-project/sglang#42145 routing small-M bf16 GEMMs to a bit-identical Triton tile to escape the DeepGEMM sm120 bandwidth cliff. Base: SGLang main 542a043 (2026-09-23), DeepGEMM 0.2.0. DFlash drafter (incoai/Qwen3.8-27B-DFlash2) at 8 draft tokens: 106.6 tok/s exact vs 109.8 non-exact vLLM short, 73.9 vs 67.4 at 2.2k, 42.6 vs 56.1 at 128k. Receipts: 24/24 token-id+logprob parity vs no-speculation (forced-reject, 8 in-flight, 128k); 0/1200 run-to-run divergences with the fence; divergences cluster where top-1/top-2 logits are within 0.25 (0.125 for backend/--max-num-seqs).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":73.900000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-27 20:30:16.519777-06","commit_sha":"d47db89f9ef31a0a96530e29d246e7f562509ee6","content_hash":"5fd4ef10f7e92f70e8b5831e5c33c58cfa0d635c","votes":0,"credit":[{"name":"bill-h-lin","url":"https://github.com/bill-h-lin/sglang-exact-spec-decode"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang.json","alternate":"/r/qwen3-8-27b-bit-exact-spec-decode-1x-rtx-pro-6000-sglang"}},{"slug":"glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn","title":"GLM-5.3-Flash VQ Q4_K 1x DGX Spark 1M ctx (JLFN)","summary":"Recipe by JLFN (dgx-recipe, verified on-box 2026-10-17). Baekpica/GLM-5.3-Flash-VQ-GGUF (72 GiB GVQ1 release directory: routed experts vector-quantized with codebook, 316 nonexpert matrices Q4_K, rest source precision; 56 GGUF parts, embedded MTP predictor) served by the ds4-dfm-rs Rust host (ds4-server, CUDA) on one DGX Spark at the model's native 1048576-token context with two banks. Measured at temperature 0: 24.2-24.3 tok/s one stream, 12.1/12.0 each with two streams (aggregate flat - decode is bytes-per-token against the weight stream). Disclosed quality trade: prose PPL 4.8555 vs 8.0797, code-completion 10.5 vs 18.4%, Korean meeting-summary degeneration under greedy.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":24.300000000000000710542735760100185871124267578125,"prefill_tok_s":0,"updated_at":"2026-10-20 21:52:36.476254-06","commit_sha":"f5a53e8b06f37327380e8affb1c9ea298e270456","content_hash":"2d63af307f133a15a2baacaee8d45c16f33ff129","votes":0,"credit":[{"name":"JLFN","url":"https://github.com/JLFN/dgx-recipe"},{"name":"Baekpica","url":"https://huggingface.co/Baekpica"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn.json","alternate":"/r/glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn"}},{"slug":"glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h","title":"GLM-5.3-Flash EXL3 3.05bpw on 1x RTX PRO 6000 - GPU expert cache, host-RAM streaming","summary":"GLM-5.3-Flash (320B, 42x288-expert MoE) served from turboderp's EXL3 3.05bpw checkpoint (116.7 GiB) on ONE 96 GB RTX PRO 6000 Blackwell: a GPU expert cache keeps the 40 most-routed experts static in VRAM plus 148 LFU cache slots on-card, and streams the remaining 100 experts per layer from 91.5 GiB of pinned host RAM by rewriting ExLlamaV3's expert pointer tables in place - stock ExLlamaV3 1.6.0 kernels and unmodified TabbyAPI (pinned commit) behind a front shim. 61.5 tok/s decode short-context, 68 tok/s at 32K-127K in context, 1.9K-2.9K tok/s prefill, 524,288-token context, native vision, MTP drafting; mean KLD 0.0680 vs the original model and 59/60 on a 60-task graded tool-use bench - roughly half the llama.cpp UD-IQ3_XXS fidelity error (0.1258) while beating it on every axis. Replaces the author's own llama.cpp recipe in the same repo.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":64.7999999999999971578290569595992565155029296875,"prefill_tok_s":2736.09999999999990905052982270717620849609375,"updated_at":"2026-10-20 15:34:15.674077-06","commit_sha":"b273a7fb8e437bb6978e08e600c43acc1fbb6fda","content_hash":"9a80e9b5da56789795695cd17f70e97159a46050","votes":0,"credit":[{"name":"SSHdotCodes","url":"https://github.com/SSHdotCodes/glm-5.3-flash-pro6000"},{"name":"turboderp","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h.json","alternate":"/r/glm-5-3-flash-exl3-3-05bpw-on-1x-rtx-pro-6000-gpu-expert-cache-h"}}]}