{"meta":{"count":27,"page":1,"pages":3,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["dgx-spark","170hx-x2","170hx-x3"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T18:30:58+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple","title":"Qwen3.8-Flash-Next W4A16 2x CMP 170HX TP2 DFlash2 host-RAM PLE","summary":"Qwen3.8-Flash-Next (125B MoE) INT4-expert W4A16 on two CMP 170HX cards, full 262,144-token context: 241/350/198 tok/s one user (structured/code/prose), 495 tok/s editing code, 895/987/770 across eight, up to 128 requests at once totaling 1,937 tok/s (DFlash 2; 1,966 MTP), 4,696 tok/s cold prefill, 1,475,836-token KV pool. Serves halt95's W4A16-Merlin checkpoint on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) with the 51 GB FP8 n-gram table in HOST RAM (cpu_offload; PLE_SHARED_DIR shares one copy across servers), the whole decode step one CUDA graph - built for a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3/2 schedule, 44% faster than MTP on code c1, 83% editing code) via engine-file patches plus RecoverSSM one-state GDN; optional KV offload restores an evicted 100k-token context in 0.89 s (21 s without). Measured 2026-10-18/23 from a fresh clone under Morrowmake's protocol (temp 0, 400-token pin, median-of-5 c1 / median-of-3 aggregates).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":241.19999999999998863131622783839702606201171875,"prefill_tok_s":0,"updated_at":"2026-10-31 11:51:46.269984-06","commit_sha":"33d1f11aee9d87ff9b04744be9ec577d953fa7c1","content_hash":"a970a21b9dad86241fc126beaaba1946d67d7b0e","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-w4a16-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"halt95 (W4A16-Merlin checkpoint + companion 3090 engine; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound INT4 experts) + RadixArk (FP8 n-gram table) + DominikBucko (MTP-head packing)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/MiaAI-Lab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple.json","alternate":"/r/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple"}},{"slug":"qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3","title":"Qwen3.8-Flash-Next-Uncensored AWQ W4A16 on 2x CMP 170HX \u2014 PP2, MTP3, PLE-from-SSD (mannydelrio)","summary":"Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE) in AWQ W4A16 (leoncca g32 build @ fa56146): PP2 with manual layer partition 25,23 (no P2P on CMP 170HX \u2014 PP beats TP, which pays a host-memory sync per layer), MTP 3-token draft, and the 29.8 GB PLE embedding table served FROM NVMe SSD via tinybeast plugins (TB_PLE_SSD=1 + prefetch so SSD reads overlap the previous GPU step). 4\u20135.7x faster than the stock TP baseline on the same cards at every prompt size, full native 262,144-token context, 897K tokens of KV. Measured by the author: 109/159 tok/s answers prose/code at 1 request (was ~37), 283/416 aggregate at 4 concurrent, prompt reading ~7,700 tok/s, TTFT 13.0 s at 100K. Quality unchanged: HumanEval 155/164, 8/8 long-doc to 222K. Engine is a SOURCE BUILD: wtdcode/vllm-backport @ 29e66dad + the repo's patches \u2014 no published image, runtime notes carry the build steps. Cards are 8GB-variant CMP 170HX unlocked to 64 GB by cmpunlocker v0.5; 180 W/card.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":109,"prefill_tok_s":7700,"updated_at":"2026-10-31 11:51:34.514899-06","commit_sha":"744dc3943504b5846fbdf2200bb5b74dd00ca990","content_hash":"902bcf143162651b991af84f818622b06504a8ab","votes":0,"credit":[{"name":"mannydelrio","url":"https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx"},{"name":"cmpunlocker (amoghmunikote)","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"clee/vllm-sm80","url":"https://github.com/clee/vllm-sm80"},{"name":"wtdcode (vllm-backport)","url":"https://github.com/wtdcode/vllm-backport"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3.json","alternate":"/r/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-pp2-mtp3"}},{"slug":"qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots","title":"Qwen3.8-Flash-Next-Uncensored AWQ W4A16 on 2x CMP 170HX \u2014 8 slots, PP2, MTP5 + int4 draft head, PLE-from-SSD (mannydelrio)","summary":"Two-card Ampere pipeline recipe for Qwen3.8-Flash-Next-Uncensored (512-expert MoE), AWQ W4A16 g32: PP2 (CMP 170HX has no P2P), MTP 5-token drafting with an int4 draft-only lm_head, 8 slots, and the 29.8 GB PLE table on NVMe. Single user 240 / 225 / 156 tok/s structured / code / prose; 8 users 1,044 / 715 / 625 aggregate; cold prefill ~6,960 tok/s; 262,144 context; HumanEval 157/164. 2x CMP 170HX unlocked to 64 GB, Gen2 x4, 180 W each.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":240.400000000000005684341886080801486968994140625,"prefill_tok_s":6964,"updated_at":"2026-10-31 11:51:33.210587-06","commit_sha":"545d29f3458cc8a926b187626f9ba5fb06c6c84f","content_hash":"abb0c007d26314948f3b53f90f638ba494e38895","votes":0,"credit":[{"name":"mannydelrio","url":"https://github.com/mannydelrio/qwen38-flash-next-2x-cmp170hx"},{"name":"AlexBrandes (W4A16 draft lm_head patch, quantizer)","url":"https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next"},{"name":"cmpunlocker (amoghmunikote)","url":"https://github.com/amoghmunikote/cmpunlocker"},{"name":"clee/vllm-sm80","url":"https://github.com/clee/vllm-sm80"},{"name":"wtdcode (vllm-backport)","url":"https://github.com/wtdcode/vllm-backport"}],"uploaded_by":"mannydelrio1","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots.json","alternate":"/r/qwen3-8-flash-next-uncensored-awq-w4a16-on-2x-cmp-170hx-8-slots"}},{"slug":"glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4","title":"GLM-5.3-Flash exlr8 EXL3 experts on DGX Spark TP1-4","summary":"GLM-5.3-Flash exlr8: MoE layers are EXL3-trellis units decoded by the custom exlr8 MoE kernel; dense linears stay FP8/NVFP4. One release carries three expert widths (k2/k3/k4 = 2/3/4 bpw), chosen at boot via VLLM_EXLR8_WIDTHS. Reference topology TP=4 on four GB10 boxes; TP=1 serves the whole model (~87 GiB) on one 128 GB DGX Spark / ASUS GX10 via tp1.yaml plus a distinct MENTAT_GROUP. TP=2 and TP=3 also measured. 2026-10-19, clocks TP-locked at 2200 MHz: decode code 128.0 tok/s, prose 72.5 at TP=4; 53.0 / 29.5 at TP=1; prefill 4,393 tok/s at 8K, 4,190 at 32K. Trade vs the shipping NVFP4 build: decode +3-8%, NLL equal or better, KV pool ~+15%, prefill 7.7% lower at Second 1x-Spark GLM-5.3-Flash lane: not the VQ Q4_K GGUF lane (/r/glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn), and not a fork of the 2x ATLAS build (/r/glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m) whose NVFP4 Flash base this stack overlays.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":65.2000000000000028421709430404007434844970703125,"prefill_tok_s":3438.5,"updated_at":"2026-10-31 11:51:31.886893-06","commit_sha":"e576a851c441c37ac2fb42dfa0188eef41f910e0","content_hash":"237f8f6570a37cb5ddb895edbae6d611ef23660e","votes":0,"credit":[{"name":"kindlingai","url":"https://github.com/kindlingai/glm-5.3-flash-exlr8-gx10"},{"name":"nvidia","url":"https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4.json","alternate":"/r/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4"}},{"slug":"glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash max effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records max reasoning effort, selected per request with model GLM-5.3-Flash-max. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":41.5,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:37.835428-06","commit_sha":"663174ab27ccbc0f93296953b897b19e08b64726","content_hash":"1bf0157fd231d61ac68175f7a1493df8e9583746","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-max-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash low effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records low reasoning effort, selected per request with model GLM-5.3-Flash-low. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":41.7000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:37.671135-06","commit_sha":"7a55561ae92fcdeb4a011950b04c4848b7666861","content_hash":"1c7b1a323fae222fd633e500b576c9f1fe1e7cec","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-low-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots","title":"GLM-5.3-Flash high effort on 2x CMP170HX EXL3 TP2 four slots","summary":"Four native active sequences on two 64 GiB CMP170HX cards; GLM-5.3-Flash EXL3 3.05bpw, TP2, with the Weightless GLP44 steering vector at alpha 2, FP16 recurrent state, Q8 GPU KV, DFlash2 K7 and hybrid15. This recipe records high reasoning effort, selected per request with model GLM-5.3-Flash-high. llm-inference-bench v0.8.0, unmodified under a per-agent counting wrapper, measures 1, 2 and 4 agents on short and nominal 32K/128K inputs, one 30-second run per cell; actual tokenized input lengths are recorded. Native completion-token usage includes reasoning. Active RAM KV paging is disabled. The 655,360-token pool is shared, with 392,960 including output as the per-request cap. Throughput is inference speed, not an accuracy or intelligence score. Differences between the low, high and max recipes are not separated from run-to-run variation, and with ignore-EOS effort changes what the model writes, not how many tokens. Accepts optional chat_template_kwargs.clear_thinking (default-off).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":43.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 23:29:26.881493-06","commit_sha":"26ce83d34683309b83e4478edb3ed1213018d87f","content_hash":"58f06fe95f6c37780466d207181753f0f7c5b12f","votes":0,"credit":[{"name":"alexvnesta (tensor-parallel fork: TP2, hybrid drafting, concurrent sessions, RAM tier, vision, NCCL backend)","url":"https://github.com/alexvnesta/glm53-flash-cmp170hx-tp"},{"name":"Flun (original 2-card layer-split server + recipe this fork starts from)","url":"https://github.com/Flun/glm53-flash-cmp170hx-exl3"},{"name":"turboderp (ExLlamaV3 engine + GLM-5.3-Flash-exl3 checkpoint + GLM-5.3 TP + vision tower support in engine)","url":"https://github.com/turboderp-org/exllamav3"},{"name":"incoai (GLM-5.3-Flash-DFlash2 BF16 drafter, CC BY-NC-ND 4.0)","url":"https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2"},{"name":"0xSero (k_hcfuse fused hyper-connection kernels, via Flun's repo)","url":"https://github.com/sybil-solutions/glm-flash-lite"},{"name":"Morrowmake (content-verified P2P check idea + benchmark prompt style)","url":"https://github.com/Morrowmake/glm53-flash-cmp170hx-recipe"},{"name":"bayley + arvkmr (fault probe idea + router-precision findings, vllm-170hx-glm5)","url":"https://github.com/bayley/vllm-170hx-glm5"},{"name":"Z.ai (GLM-5.3-Flash base model)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"msuiche (Weightless GLP-44 vector and specification)","url":"https://huggingface.co/msuiche/GLM-5.3-Flash-abliterated-cyber-GLP-44"}],"uploaded_by":"alex.v.nesta","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots.json","alternate":"/r/glm-5-3-flash-high-effort-on-2x-cmp170hx-exl3-tp2-four-slots"}},{"slug":"qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3","title":"Qwen3.8-Flash-Next W4A16 INT4PLE on 2x modded CMP 170HX \u2014 PP2 MTP3 (DUFRelic)","summary":"Production 2x CMP 170HX recipe for Qwen3.8-Flash-Next: zankich INT4PLE repackage of halt95 Merlin W4A16 (94.4 GiB, 51B-param PLE table repacked INT4) on stock vLLM 0.30.0 + a 6-layer hash-guarded overlay stack (PP2/PLE #56444, KV filter #54793, MTP-over-PP #54709, slot statefix #55506, SpeedSuite 5-PR overlay, batched-P2P). PP2/TP1, MTP 3-token draft (measured 52.27% accept). Hardware is the recipe's own mod stack: 74-SM unlock (cmpunlocker), VBIOS 92.00.6D.00.0A, 170tune 1420 MHz SM cap + HBM 1728 MHz undervolt, BAR1 P2P driver patches, 140 W/card. Author-measured 2026-10-03 (median of 3, isolation audit): prefill C1 6213.7 tok/s @16k, decode C1 133.4 tok/s (SD 22.8 reported), decode C4 agg 245.2 tok/s; 55/56 C at 160 W peaks, no Xid/OOM. INT4 PLE table (~26.8 GiB) pinned in HOST RAM; KV BF16 (pinned build's QSA path rejects FP8). Driver mods optional for function but required for the published throughput.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":133.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:28.370851-06","commit_sha":"c26a7fd62ded418a43f5a10abf82eaedb7afc6bc","content_hash":"7c6051a80e2a07b8bca384eb300935641c88349b","votes":0,"credit":[{"name":"DUFRelic","url":"https://github.com/DUFRelic/qwen38-flash-next-recipe"},{"name":"zankich (INT4-PLE repackage + plugin)","url":"https://github.com/zankich/vllm"},{"name":"halt95 (W4A16-Merlin base checkpoint)","url":"https://github.com/halt95/merlin-engine"},{"name":"michael-lgtm (public overlay rebuild)","url":"https://github.com/michael-lgtm/qwen38-flash-next-cmp-overlay"},{"name":"amoghmunikote (cmpunlocker SM unlock + BAR1 P2P)","url":"https://github.com/amoghmunikote/cmpunlocker"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3.json","alternate":"/r/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3"}},{"slug":"qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g","title":"Qwen3.8-Flash-Next W4A16-AutoRound DGX Spark vLLM MTP4 Iter7g","summary":"Recipe by kyaky. Qwen3.8-Flash-Next W4A16-AutoRound hybrid (INT4 experts + FP8 side layers + INT8 head; 51B-token n-gram PLE table on NVMe mmap) on one DGX Spark (GB10, 121 GB unified memory) on a verified-deterministic vLLM floor (image qwen38-flash-dgx:iter7g-20261015 on the digest-pinned vllm/vllm-openai:qwen38-flash-next base): MoE align arrival-order shim, drafter-Marlin atomicAdd fix, pinned triton config, compile-cache mount. Measured line v19a-w5 = MTP=4 window-5 block verification: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 streams (+9.2% N1 / +4.8% N8 over MTP=3, two boots per arm, same image). Promoted serving line v20-fp8: fp8 KV cache on a 24 GB pool at native 262,144 context, 1,291,120-token capacity. Chinese +31.3% via a 25,862-id (39.5%) Han draft-vocab rebalance (prose-zhen 34.8 to 45.7 tok/s). Stochastic-correctness gate 37/37 vs bit-exact fp64 oracle; per-round raw data + scripts ship in-repo; copy-agentic-1 failure disclosed.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":61.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:28.107205-06","commit_sha":"dc8982acabf5e6b6417f4871ad5149d61f01755d","content_hash":"e6a2cfdf4903c7ca1b80df3d0f496d3ca3f13840","votes":0,"credit":[{"name":"kyaky","url":"https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g.json","alternate":"/r/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g"}},{"slug":"qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp","title":"Qwen3.8-Flash-Next NVFP4 DGX Spark GB10 SGLang NEXTN MTP","summary":"Production single-node DGX Spark (Grace Blackwell GB10, SM121, 128GB unified memory) recipe for Qwen3.8-Flash-Next on SGLang: ModelOpt NVFP4 quantization with zero-copy NVMe mmap of the 47.7 GiB ple_table.bin via --ple-offload-embedding (eliminates 51 GiB of unified-memory footprint), custom Triton FP4 prefill kernels + trtllm_mha decode kernels on SM121, NEXTN MTP (3 verification steps / 4 draft tokens) lifting output to 110-152 tok/s, GDN+QSA hybrid (131,072-token served window, 262k native capacity), 96-slot mamba recurrence buffers, sharp froggeric v22.1 Jinja chat template (-18% response length), persistent Triton/Inductor SM121 JIT caches on NVMe, 8-worker parallel NVMe pre-warm (206 shards in 17s) + 16-core Grace OMP. Author-measured vs same-box 27B dense baseline: 110.4-152.8 tok/s decode w/ MTP (baseline 12.1 base / 21.5 MTP), single-stream CoT 25.1-28.8 tok/s (2.62x speculative gain), TTFT ~347ms radix hit / ~465ms cold, 113 GiB / 121 GiB RAM used (~8 GiB headroom), 103.5 GiB","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":110.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:27.869274-06","commit_sha":"2c5220aa85b5379d7373abd0aa14bb13b1ec993f","content_hash":"bf0c02f95b99494d6720a515e2f8f438f126bc87","votes":0,"credit":[{"name":"Michael Bemler (Bemler Labs)","url":"https://github.com/bemlerlabs/qwen3.8-flash-next-dgx-spark-sglang"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp.json","alternate":"/r/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp"}},{"slug":"qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6","title":"Qwen3.8-Flash-Next NVFP4 2x CMP 170HX TEP2 GDS SSD PLE MTP6","summary":"Recipe by nguyenthimy2022kg-alt. Community patched vLLM run of the NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on 2x unlocked CMP 170HX (SM80, ~63.4 GiB usable each) with TEP2 tensor+expert parallel and fixed MTP6 speculative decoding with per-request draft isolation. v0.3.0 supports 1-10 active requests (max_num_seqs accepts 1-10, defaults to 1 when omitted; repo example sets 10) with compact/topk vocabulary transport and expanded PCIe communication; history-copy (ngram hybrid) drafting removed. The ~51.2 GB FP8 PLE table stays on NVMe, read GPU-direct via GDS/cuFile with 32 persistent readers and 3 staging slots - no CPU-memory relay; optional host-read PLE backend off by default. FlashInfer PCIe IPC AllReduce, 36-token prefix matching, bayley/cmpunlocker BAR1/P2P driver + host GDS. Measured upstream (2026-10-11, thinking off): 10 requests x 5,000 tokens, 749.67 tok/s aggregate, 76.29/request; historical 2026-09-08 window 151.05-168.29 tok/s decode at 8K-128K inputs xhigh thinking.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":158.19999999999998863131622783839702606201171875,"prefill_tok_s":2059.90000000000009094947017729282379150390625,"updated_at":"2026-10-29 21:32:27.639988-06","commit_sha":"7d911c7dd2ce6b0a45d3c14791ee74036a43e7ae","content_hash":"247ee93d0bf55ac542edef7091844ac3df34a323","votes":0,"credit":[{"name":"nguyenthimy2022kg-alt","url":"https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6.json","alternate":"/r/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6"}},{"slug":"qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x","title":"Qwen3.8-Flash-Next FP4-Local 1x DGX Spark vLLM b12x","summary":"Recipe by sojufx. Serves NVIDIA's nvidia/Qwen3.8-Flash-Next-FP4-Local (96 GiB NVFP4 quantization-aware-distilled checkpoint in a new compressed NVFP4 scale format) on one DGX Spark (GB10) via the digest-pinned eugr/spark-vllm-b12x community container (reported vLLM 0.1.dev21830+g8244d1c44.d20261015) - the model card requires community b12x containers until the format lands upstream. FP8 KV cache, exact MTP K=3 with standard rejection sampling, 262,144-token context, 4 scheduler slots, 4096-token batched prefill, disk-backed io_uring PLE, prefix cache with Mamba align, vision plus qwen3_xml tool calling, thinking off by default, localhost-only 127.0.0.1:8001 OpenAI-compatible backend with a deliberate no-replace/no-port-take guard. Measured 2026-10-15/16 on the author's Spark: 66.2 tok/s single-stream sparkDash counting decode, 166.4 tok/s at C4, 41-54 tok/s single-stream on the four-prompt wall-clock suite - different measurement bases, not interchangeable.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":66.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:27.177524-06","commit_sha":"5fc0d101d8e9b1c6a1419f3d465e21649f4ecbad","content_hash":"be42677d0cc5ca7f71fe72210e756f1622fe3604","votes":0,"credit":[{"name":"sojufx","url":"https://github.com/sojufx/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark-Recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x.json","alternate":"/r/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x"}}]}