{"meta":{"count":14,"page":1,"pages":2,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["dgx-spark","170hx","170hx-x6"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T23:34:54+00:00"},"recipes":[{"slug":"glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4","title":"GLM-5.3-Flash exlr8 EXL3 experts on DGX Spark TP1-4","summary":"GLM-5.3-Flash exlr8: MoE layers are EXL3-trellis units decoded by the custom exlr8 MoE kernel; dense linears stay FP8/NVFP4. One release carries three expert widths (k2/k3/k4 = 2/3/4 bpw), chosen at boot via VLLM_EXLR8_WIDTHS. Reference topology TP=4 on four GB10 boxes; TP=1 serves the whole model (~87 GiB) on one 128 GB DGX Spark / ASUS GX10 via tp1.yaml plus a distinct MENTAT_GROUP. TP=2 and TP=3 also measured. 2026-10-19, clocks TP-locked at 2200 MHz: decode code 128.0 tok/s, prose 72.5 at TP=4; 53.0 / 29.5 at TP=1; prefill 4,393 tok/s at 8K, 4,190 at 32K. Trade vs the shipping NVFP4 build: decode +3-8%, NLL equal or better, KV pool ~+15%, prefill 7.7% lower at Second 1x-Spark GLM-5.3-Flash lane: not the VQ Q4_K GGUF lane (/r/glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn), and not a fork of the 2x ATLAS build (/r/glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m) whose NVFP4 Flash base this stack overlays.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":65.2000000000000028421709430404007434844970703125,"prefill_tok_s":3438.5,"updated_at":"2026-10-31 11:51:31.886893-06","commit_sha":"e576a851c441c37ac2fb42dfa0188eef41f910e0","content_hash":"237f8f6570a37cb5ddb895edbae6d611ef23660e","votes":0,"credit":[{"name":"kindlingai","url":"https://github.com/kindlingai/glm-5.3-flash-exlr8-gx10"},{"name":"nvidia","url":"https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4.json","alternate":"/r/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4"}},{"slug":"qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g","title":"Qwen3.8-Flash-Next W4A16-AutoRound DGX Spark vLLM MTP4 Iter7g","summary":"Recipe by kyaky. Qwen3.8-Flash-Next W4A16-AutoRound hybrid (INT4 experts + FP8 side layers + INT8 head; 51B-token n-gram PLE table on NVMe mmap) on one DGX Spark (GB10, 121 GB unified memory) on a verified-deterministic vLLM floor (image qwen38-flash-dgx:iter7g-20261015 on the digest-pinned vllm/vllm-openai:qwen38-flash-next base): MoE align arrival-order shim, drafter-Marlin atomicAdd fix, pinned triton config, compile-cache mount. Measured line v19a-w5 = MTP=4 window-5 block verification: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 streams (+9.2% N1 / +4.8% N8 over MTP=3, two boots per arm, same image). Promoted serving line v20-fp8: fp8 KV cache on a 24 GB pool at native 262,144 context, 1,291,120-token capacity. Chinese +31.3% via a 25,862-id (39.5%) Han draft-vocab rebalance (prose-zhen 34.8 to 45.7 tok/s). Stochastic-correctness gate 37/37 vs bit-exact fp64 oracle; per-round raw data + scripts ship in-repo; copy-agentic-1 failure disclosed.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":61.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:28.107205-06","commit_sha":"dc8982acabf5e6b6417f4871ad5149d61f01755d","content_hash":"e6a2cfdf4903c7ca1b80df3d0f496d3ca3f13840","votes":0,"credit":[{"name":"kyaky","url":"https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g.json","alternate":"/r/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g"}},{"slug":"qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp","title":"Qwen3.8-Flash-Next NVFP4 DGX Spark GB10 SGLang NEXTN MTP","summary":"Production single-node DGX Spark (Grace Blackwell GB10, SM121, 128GB unified memory) recipe for Qwen3.8-Flash-Next on SGLang: ModelOpt NVFP4 quantization with zero-copy NVMe mmap of the 47.7 GiB ple_table.bin via --ple-offload-embedding (eliminates 51 GiB of unified-memory footprint), custom Triton FP4 prefill kernels + trtllm_mha decode kernels on SM121, NEXTN MTP (3 verification steps / 4 draft tokens) lifting output to 110-152 tok/s, GDN+QSA hybrid (131,072-token served window, 262k native capacity), 96-slot mamba recurrence buffers, sharp froggeric v22.1 Jinja chat template (-18% response length), persistent Triton/Inductor SM121 JIT caches on NVMe, 8-worker parallel NVMe pre-warm (206 shards in 17s) + 16-core Grace OMP. Author-measured vs same-box 27B dense baseline: 110.4-152.8 tok/s decode w/ MTP (baseline 12.1 base / 21.5 MTP), single-stream CoT 25.1-28.8 tok/s (2.62x speculative gain), TTFT ~347ms radix hit / ~465ms cold, 113 GiB / 121 GiB RAM used (~8 GiB headroom), 103.5 GiB","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":110.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:27.869274-06","commit_sha":"2c5220aa85b5379d7373abd0aa14bb13b1ec993f","content_hash":"bf0c02f95b99494d6720a515e2f8f438f126bc87","votes":0,"credit":[{"name":"Michael Bemler (Bemler Labs)","url":"https://github.com/bemlerlabs/qwen3.8-flash-next-dgx-spark-sglang"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp.json","alternate":"/r/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp"}},{"slug":"qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x","title":"Qwen3.8-Flash-Next FP4-Local 1x DGX Spark vLLM b12x","summary":"Recipe by sojufx. Serves NVIDIA's nvidia/Qwen3.8-Flash-Next-FP4-Local (96 GiB NVFP4 quantization-aware-distilled checkpoint in a new compressed NVFP4 scale format) on one DGX Spark (GB10) via the digest-pinned eugr/spark-vllm-b12x community container (reported vLLM 0.1.dev21830+g8244d1c44.d20261015) - the model card requires community b12x containers until the format lands upstream. FP8 KV cache, exact MTP K=3 with standard rejection sampling, 262,144-token context, 4 scheduler slots, 4096-token batched prefill, disk-backed io_uring PLE, prefix cache with Mamba align, vision plus qwen3_xml tool calling, thinking off by default, localhost-only 127.0.0.1:8001 OpenAI-compatible backend with a deliberate no-replace/no-port-take guard. Measured 2026-10-15/16 on the author's Spark: 66.2 tok/s single-stream sparkDash counting decode, 166.4 tok/s at C4, 41-54 tok/s single-stream on the four-prompt wall-clock suite - different measurement bases, not interchangeable.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":66.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:27.177524-06","commit_sha":"5fc0d101d8e9b1c6a1419f3d465e21649f4ecbad","content_hash":"be42677d0cc5ca7f71fe72210e756f1622fe3604","votes":0,"credit":[{"name":"sojufx","url":"https://github.com/sojufx/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark-Recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x.json","alternate":"/r/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x"}},{"slug":"qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm","title":"Qwen3.8-Flash-Next 3bpw FP8-PLE 1x CMP 170HX DFlash2 RecoverSSM","summary":"Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s editing code, 799.5/791.3/586.4 across eight, up to 48 requests at once - 1,384 tok/s total with the default DFlash2 drafter (1,403 with the checkpoint's MTP head), 4,243 tok/s cold prefill, 292,784-token KV pool. Serves klee100's 3 bpw AutoRound checkpoint (3-bit g128 / 2-bit g64 experts, 47 GiB on card) with halt95's FP8 n-gram table in HOST RAM (48 GiB, hard-linked at build) on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) so the whole decode step is one CUDA graph on a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3 by request count) + the recipe's RecoverSSM + BF16 GDN-state patches (9 files mounted over the image) fit 48 concurrent requests on one card. Measured 2026-10-23 under the MiaAI-Lab protocol.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":197.80000000000001136868377216160297393798828125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.955646-06","commit_sha":"af7d7dde12376551476c8b9e36cd72c68dc32408","content_hash":"c82e80ffa9955d905b61bda381cf7d89db1350ba","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/Mia-AiLab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json","alternate":"/r/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm"}},{"slug":"nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k","title":"Nemotron 3.5 Bolt 74B-A9B 1x DGX Spark vLLM b12x LiLiCorr 262K","summary":"NVIDIA Nemotron 3.5 Bolt 74B-A9B-24GB (hybrid Mamba-2/Transformer latent-MoE, 74B with 9B active, IQ2_XXS 2-bit experts, NVFP4 Mamba layers, Q8_0 rest; text+image+video in, 262,144-token window, OpenMDW-1.1) on one DGX Spark (GB10/SM121) with vLLM nightly-77154ca6 (base image vllm/vllm-openai@sha256:1ee85ca8) + b12x 1.5.1 kernels (--linear-backend b12x --moe-backend b12x), FP8 KV cache, gpu-memory-utilization 0.70, max-num-seqs 16, and the NVIDIA LiLiCorr MTP drafter at 7 speculative tokens. Pinned revisions: model c6cc2e8f, drafter 4c6a5613. Measured on the reference box: ~140 tok/s code rewrite / 63 summarise / 50 prose single-stream with the drafter (mean acceptance 4.0 tokens/pass), 31 tok/s without; 74.8/113.4/163.6/201.2 tok/s at 1/4/8/16 streams; KV cache ~6.52M tokens at the 0.70 share. On a 64 GB Spark the lane takes its own memory share 0.60: 667,874 KV tokens (2.55 windows), 110.8 tok/s greedy, 7.1 GiB left at the peak (2026-10-24).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":63,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.717952-06","commit_sha":"2cdb5f11ffa1fb1af8676496fdd631976c3d536d","content_hash":"28887fb2cf3bab1914c803e792898fff92b9afe0","votes":0,"credit":[{"name":"hasso5703","url":"https://github.com/hasso5703"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k.json","alternate":"/r/nemotron-3-5-bolt-74b-a9b-1x-dgx-spark-vllm-b12x-lilicorr-262k"}},{"slug":"qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic","title":"Qwen3.8-27B SGLang NVFP4 DFlash2 on 1x DGX Spark (deterministic)","summary":"First dense Qwen3.8-27B recipe on 1x DGX Spark (GB10, 128 GB unified): SGLang v0.5.22 release image + NVFP4 target + DFlash2 speculative decoding (NVFP4 calibrated 5-layer draft, depth 16), served by a boot-persistent systemd unit from a one-command installer. Author-measured 68.3 tok/s greedy median single stream (bench.sh, thinking on; v1.39.0 A/B 2026-10-23), 81.7 tok/s at 4 streams (serving-bench), 135-148 tok/s at 8 streams and 258 at 32. Fully pinned (image digest, checkpoint revisions, draft depth) and reproducible to the decimal across boots via --disable-flashinfer-autotune (\"The boot lottery\", BENCHMARKS.md); 1M-token YaRN context by default, native 262144 as fallback; OpenAI- and Anthropic-compatible API on :30001 through the box's keepalive proxy (the engine itself answers on 127.0.0.1:30000, on the box only), with opencode wired out of the box.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":68.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-10-28 16:49:58.824279-06","commit_sha":"307fa2334d111e38258e27fa9e690c44e96b441b","content_hash":"7d8b3e291e1b8abfa9db4ded3e90cd601b8bba78","votes":0,"credit":[{"name":"hasso5703","url":"https://github.com/hasso5703"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic.json","alternate":"/r/qwen3-8-27b-sglang-nvfp4-dflash2-on-1x-dgx-spark-deterministic"}},{"slug":"qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data","title":"Qwen3.8-Flash-Next NVFP4 1x DGX Spark vLLM 71.3 c1 (salient-data)","summary":"Qwen3.8-Flash-Next (~180B multimodal MoE, NVFP4) on one NVIDIA DGX Spark (GB10, 121 GiB unified memory) with a pinned vLLM nightly image + 14 provenance-tracked patches (patch --fuzz=0) + one docker run, fitting the 51.2 GB PLE n-gram table by never making it resident (rows read per forward-pass step from the checkpoint file). 69.3 tok/s single-stream median decode (runs 66.8/69.5/69.3), 41.0 prose median, 91.9 tok/s 6-stream aggregate (1 run), GSM8K 99/100 on a fixed 100-problem subset (temp 0.6 top-p 0.95 thinking on), 12/12 arithmetic + 131K needle pass + prefix-cache canary pass, prefix caching ON. Measured with tonyd2wild's bench_categories.py @ 6ad1c8f unchanged on build 2026-10-22-8e2beaa3; clean first-time run 2026-10-25 re-measured 69.8/41.5/82.6. Prod/dev: prod image digest-pinned + cosign-signed + SPDX SBOM; dev rebuilds the same serving code from published patches with an AST-equivalence proof. Outside the model card's support statement (B200/B300 listed, GB10 unlisted).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":69.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-10-28 15:59:39.152527-06","commit_sha":"85e2e4120ef69829bff82736c39745ac579c9a6b","content_hash":"63f86642d07e8833929005e2087044adaecc1ca4","votes":0,"credit":[{"name":"Jeremy Newhouse / salient-data (recipe, patches, measurements)","url":"https://github.com/salient-data/qwen38-next-dgx-recipe"},{"name":"tonyd2wild (bench_categories.py @ 6ad1c8f + comparator stack/numbers)","url":"https://github.com/tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark"},{"name":"NVIDIA (Qwen3.8-Flash-Next-NVFP4 checkpoint; HF token gated)","url":"https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4"},{"name":"vLLM project (nightly base image 8e2beaa3)","url":"https://github.com/vllm-project/vllm"},{"name":"Sigstore/cosign (image signing + SBOM timestamping)","url":"https://docs.sigstore.dev"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data.json","alternate":"/r/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data"}},{"slug":"deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5","title":"DeepSeek-V4.1-Flash SM80 DeepInfer kernel port 6x CMP 170HX DSpark K5","summary":"DeepSeek-V4.1-Flash served on six 64 GB SM80 GPUs (GA100 die, owner-classified 170HX) with HalfVulpes' Deepinfer-SM80 patch set: a 30-file vLLM V1 overlay porting DeepSeek's DeepInfer kernel architecture (SM100 tcgen05/TMEM) to Ampere mma.sync tensor cores - unified 512-dim SM80 sparse-MLA decode/prefill kernels with bitshift UE8M0 scale decoding, a bit-exact single-CTA Triton TopK+SWA combiner (22.17x), direct-store KV-cache gather (7.64x @48K), SM80-aligned TileLang mHC routing (2.89x), and YOCO L20-L39 ragged indexer-metadata reuse (19 redundant kernel chains eliminated per chunk). TP2 x PP3 (14/14/12) + DSpark K=5 + 1M context, 4.65M-token GPU-resident KV. Author-measured: 64K cold prefill 4,507 tok/s (+24.2% vs baseline vLLM SM80), 64K TTFT 10.7 s (-18.9%), c1 DSpark decode 160.8 tok/s code / 188.8 tok/s 16K+ long-gen, c4 170.5 aggregate. Microbenchmarks bit-exact (0 diff) or within BF16 noise. Topology guide: PP6 = 2.1x slower single-stream; TP2+TP4 asymmetric hurts Marlin tiles.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":160.80000000000001136868377216160297393798828125,"prefill_tok_s":4506.8000000000001818989403545856475830078125,"updated_at":"2026-10-28 15:48:17.404025-06","commit_sha":"ced28259b90a43ec103ecd450b68455a56d2808d","content_hash":"e00265eca8b84734c2c7de2b67d558b0d702d25e","votes":0,"credit":[{"name":"HalfVulpes (Deepinfer-SM80 port + measurements)","url":"https://github.com/HalfVulpes/Deepinfer-SM80"},{"name":"DeepSeek-AI (DeepInfer engine architecture + DeepSeek-V4.1-Flash checkpoint)","url":"https://github.com/deepseek-ai/DeepInfer"},{"name":"vLLM project (V1 engine + TPxPP runtime)","url":"https://github.com/vllm-project/vllm"},{"name":"TileLang + OpenAI Triton (SM80 kernel synthesis)","url":"https://github.com/tile-ai/tilelang"},{"name":"FlashInfer + IST-DASLab Marlin (sparse-MLA + MXFP4/W4A16 Ampere GEMM primitives)","url":"https://github.com/flashinfer-ai/flashinfer"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5.json","alternate":"/r/deepseek-v4-1-flash-sm80-deepinfer-kernel-port-6x-a100-dspark-k5"}},{"slug":"glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn","title":"GLM-5.3-Flash VQ Q4_K 1x DGX Spark 1M ctx (JLFN)","summary":"Recipe by JLFN (dgx-recipe, verified on-box 2026-10-17). Baekpica/GLM-5.3-Flash-VQ-GGUF (72 GiB GVQ1 release directory: routed experts vector-quantized with codebook, 316 nonexpert matrices Q4_K, rest source precision; 56 GGUF parts, embedded MTP predictor) served by the ds4-dfm-rs Rust host (ds4-server, CUDA) on one DGX Spark at the model's native 1048576-token context with two banks. Measured at temperature 0: 24.2-24.3 tok/s one stream, 12.1/12.0 each with two streams (aggregate flat - decode is bytes-per-token against the weight stream). Disclosed quality trade: prose PPL 4.8555 vs 8.0797, code-completion 10.5 vs 18.4%, Korean meeting-summary degeneration under greedy.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":24.300000000000000710542735760100185871124267578125,"prefill_tok_s":0,"updated_at":"2026-10-20 21:52:36.476254-06","commit_sha":"f5a53e8b06f37327380e8affb1c9ea298e270456","content_hash":"2d63af307f133a15a2baacaee8d45c16f33ff129","votes":0,"credit":[{"name":"JLFN","url":"https://github.com/JLFN/dgx-recipe"},{"name":"Baekpica","url":"https://huggingface.co/Baekpica"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn.json","alternate":"/r/glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn"}},{"slug":"qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix","title":"Qwen3.8-27B INT8 MTP3 prefix-cache on 1x CMP 170HX Hynix","summary":"Qwen3.8-27B at 65-67 tok/s decode (74 structured) with MTP k=3 speculative decoding AND prefix caching simultaneously on a single NVIDIA CMP 170HX 64GB (Hynix HBM2e, GA100/sm80), 262,144-token context on fp8_e4m3 KV, 24/7 production on pinned upstream vLLM nightly 0.26.1rc1.dev1024+g1fe3a1571 \u2014 the first build where prefix caching and MTP spec decode coexist (GDN mamba-align). Cached TTFT on a 7K shared prefix drops ~4.0 s -> 1.44 s (33.5% hit rate) while decode gains 7-9% over the 0.27.1 fork fallback \u2014 the 'MTP vs prefix cache, pick one' tradeoff is gone. Load-bearing trio --async-scheduling + --max-num-batched-tokens 8192 + --max-num-scheduled-tokens 8192 (default chunked-prefill batch = 0.0% hit rate while looking healthy); --prefix-match-unit 400 halves cached TTFT further (must divide the 1600-token GDN block); batch budget 32000 OOMs and hard-wedges the card. Samsung-40GB side recipe (int8 + DFlash2, 97.8 tok/s) with an sm80 int8-repack CPU-fallback patch.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":68.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-20 15:34:22.585629-06","commit_sha":"ad3f55dab02cedfb2b187f7dcfc3f1647e0aaa0c","content_hash":"68f95048b88c132d1ddbffc8ceceaf5c38be5140","votes":0,"credit":[{"name":"ahnguyen17 (recipe, patches, probes, A/B benchmarks)","url":"https://github.com/ahnguyen17/cmp-170hx-vllm"},{"name":"lued (Qwen3.8-27B-INT8-W8A16-MTP checkpoint)","url":"https://huggingface.co/lued/Qwen3.8-27B-INT8-W8A16-MTP"},{"name":"vllm-project (vLLM upstream; #50021 GDN spec bounds, #45238 align-mode cache retention)","url":"https://github.com/vllm-project/vllm"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix.json","alternate":"/r/qwen3-8-27b-int8-mtp3-prefix-cache-on-1x-cmp-170hx-hynix"}},{"slug":"qwen3-8-flash-next-uncensored-nvfp4-bf16-ple-dgx-spark-vllm-mtp2","title":"Qwen3.8-Flash-Next Uncensored NVFP4 BF16 PLE DGX Spark vLLM MTP2","summary":"DGX Spark recipe running NVFP4 Qwen3.8-Flash-Next with its FULL-PRECISION BF16 PLE memory-mapped from NVMe on vLLM (NOT the FP8-table variants the other Spark recipes use): the complete 95.37 GiB BF16 PLE stays in the checkpoint on NVMe and is mmap'd, gathering only the rows each request needs, while the NVFP4 compute trunk stays in unified memory - PIECEWISE CUDA graphs + BF16 KV cache. Runtime-qualified ONLY for orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 @ 3a3b63161c0745390e5270179af42e46efc70799 (gated HF repo) plus the recipe's deterministic overlay of exactly 31 native BF16 MTP tensors sourced from pinned RadixArk/Qwen3.8-Flash-Next-NVFP4 @ 7b719225242aacd3dbd3f9407468c2ee9a9d2594 (Orcarouter ships zero MTP tensors despite advertising an MTP layer; 4.86 GiB compact BF16 MTP file; load guard verifies 31/31 tensor identities, hash+dtype+name+arch+load-count guarded - checkpoint-specific, not a generic plug-in). Engine: official vllm/vllm-openai:qwen38-flash-next image @ sha256:f","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":44.2000000000000028421709430404007434844970703125,"prefill_tok_s":2055.1999999999998181010596454143524169921875,"updated_at":"2026-10-20 15:34:13.655428-06","commit_sha":"fbfeac5f3b0f82db6cddd8164911dcb3173edf1d","content_hash":"89ee2463a76211f23776cd44e1d72def5dd56dc5","votes":0,"credit":[{"name":"YSLAB-ai (BF16 PLE support, DGX Spark fixes, guarded MTP overlay, runtime qualification)","url":"https://github.com/YSLAB-ai/Qwen3.8-Flash-Next-NVFP4-BF16PLE-DGX-Spark"},{"name":"blazux (originated the FP8 PLE mmap work)","url":"https://github.com/blazux"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-uncensored-nvfp4-bf16-ple-dgx-spark-vllm-mtp2.json","alternate":"/r/qwen3-8-flash-next-uncensored-nvfp4-bf16-ple-dgx-spark-vllm-mtp2"}}]}