{"meta":{"count":26,"page":1,"pages":3,"per_page":12,"applied":{"engine":null,"model":"qwen3-8-flash-next","quant":null,"storage":null,"hardware":null,"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T06:50:28+00:00"},"recipes":[{"slug":"qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3","title":"Qwen3.8-Flash-Next NVFP4 on RTX PRO 6000 Blackwell \u2014 TP1 MTP3","summary":"Single-card Blackwell recipe for Qwen3.8-Flash-Next (hybrid GDN/QSA MoE, 48 layers, 512 experts) on one 96 GB RTX PRO 6000: the Community image community image R34 (localinferencelab/vllm, v0.26.1rc0+glm53.r34), TP1 with MTP3 speculative decoding (b12x MoE backend), 256k context. Benchmark-qualified on RTX PRO 6000 Blackwell Max-Q: decode 148 tok/s at ctx-0 (temp 1), prefill 3860 tok/s at 2048 batched tokens. Launch argv transcribed verbatim from the r35 community deployment. Weights: JMNI-Labs/Qwen3.8-Flash-Next-NVFP4-QAD5500-Hybrid \u2014 step-5500 QAD trained tensors recombined with main 7c4f1bc1 MXFP8 attention + NVFP4 MTP experts; same-stack measured +5% MTP-normalized decode, +25% with MTP vs the main checkpoint (evals within noise, MMLU-Pro 66.9 vs 67.2).","tier":"curated","beta":false,"generation":"blackwell","decode_tok_s":137.19999999999998863131622783839702606201171875,"prefill_tok_s":6627.3000000000001818989403545856475830078125,"updated_at":"2026-10-29 23:29:38.177885-06","commit_sha":"28709ecba026aff18371426bc26025e5a4185209","content_hash":"1e37b0f978392ba766c01cfdf9ae9ee3e002ee84","votes":1,"credit":[{"name":"local-inference-lab","url":"https://github.com/local-inference-lab/rtx6kpro"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3.json","alternate":"/r/qwen3-8-flash-next-nvfp4-1x6000-jj-mtp3"}},{"slug":"qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f","title":"Qwen3.8-Flash-Next W4A16-Merlin on 4x RTX 3090 \u2014 TP2xPP2+EP, MTP3, FP8 KV, 925K pool (halt95)","summary":"Four-card RTX 3090 recipe for Qwen3.8-Flash-Next (125B MoE) in halt95's W4A16-Merlin build (Intel AutoRound INT4 + FP8 PLE + MTP draft, @ 17284bf): TP2 x PP2 with expert parallel, manual layer partition 25,23, FP8 KV with a 924,993-token pool (three full 262K sessions resident; block-size 4096, prefix-match-unit 64), MTP 3-token probabilistic draft with RecoverSSM, host-mapped PLE pull transport, host-resident embeddings, side-cache layout + Model Runner V2, 42 images/request. Agent-focused: 92.5% of follow-up prompt tokens from cache across 8 sessions, median follow-up first token 1.61 s (0.96 s solo). Maintainer-measured (submitter T0, v2.5.1 bench card 2026-10-06): 5,219-5,533 tok/s cold prefill 10K-261K, 158-174 tok/s single-stream decode (thinking on), 534 tok/s aggregate at 8 concurrent, 262K needle exact, GSM8K 198/200. Engine is a SOURCE BUILD: vLLM v0.30.0 + 97 commits (release bundle verified by commit AND tree hash); argv verbatim from scripts/serve-v2.5.sh @ 28cbb72.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":168.099999999999994315658113919198513031005859375,"prefill_tok_s":5360,"updated_at":"2026-10-29 21:32:28.596454-06","commit_sha":"d2343f0bc9d171b68356492709d3915f2e0eb610","content_hash":"8aefc67a1e986e515743a30617a02bbc169edc71","votes":0,"credit":[{"name":"halt95","url":"https://github.com/halt95/merlin-engine"},{"name":"Qwen (Qwen3.8-Flash-Next)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"Intel AutoRound (INT4 experts)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"RadixArk (FP8 PLE table)","url":"https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4"},{"name":"DominikBucko (MTP INT4 packing recipe)","url":"https://github.com/DominikBucko/qwen38-flash-next-2x3090"},{"name":"peakcrosser7 (Flash-Next vLLM branch)","url":"https://github.com/peakcrosser7/vllm/commits/release/qwen38next_offload"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f.json","alternate":"/r/qwen3-8-flash-next-w4a16-merlin-on-4x-rtx-3090-tp2xpp2-ep-mtp3-f"}},{"slug":"qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3","title":"Qwen3.8-Flash-Next W4A16 INT4PLE on 2x modded CMP 170HX \u2014 PP2 MTP3 (DUFRelic)","summary":"Production 2x CMP 170HX recipe for Qwen3.8-Flash-Next: zankich INT4PLE repackage of halt95 Merlin W4A16 (94.4 GiB, 51B-param PLE table repacked INT4) on stock vLLM 0.30.0 + a 6-layer hash-guarded overlay stack (PP2/PLE #56444, KV filter #54793, MTP-over-PP #54709, slot statefix #55506, SpeedSuite 5-PR overlay, batched-P2P). PP2/TP1, MTP 3-token draft (measured 52.27% accept). Hardware is the recipe's own mod stack: 74-SM unlock (cmpunlocker), VBIOS 92.00.6D.00.0A, 170tune 1420 MHz SM cap + HBM 1728 MHz undervolt, BAR1 P2P driver patches, 140 W/card. Author-measured 2026-10-03 (median of 3, isolation audit): prefill C1 6213.7 tok/s @16k, decode C1 133.4 tok/s (SD 22.8 reported), decode C4 agg 245.2 tok/s; 55/56 C at 160 W peaks, no Xid/OOM. INT4 PLE table (~26.8 GiB) pinned in HOST RAM; KV BF16 (pinned build's QSA path rejects FP8). Driver mods optional for function but required for the published throughput.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":133.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:28.370851-06","commit_sha":"c26a7fd62ded418a43f5a10abf82eaedb7afc6bc","content_hash":"7c6051a80e2a07b8bca384eb300935641c88349b","votes":0,"credit":[{"name":"DUFRelic","url":"https://github.com/DUFRelic/qwen38-flash-next-recipe"},{"name":"zankich (INT4-PLE repackage + plugin)","url":"https://github.com/zankich/vllm"},{"name":"halt95 (W4A16-Merlin base checkpoint)","url":"https://github.com/halt95/merlin-engine"},{"name":"michael-lgtm (public overlay rebuild)","url":"https://github.com/michael-lgtm/qwen38-flash-next-cmp-overlay"},{"name":"amoghmunikote (cmpunlocker SM unlock + BAR1 P2P)","url":"https://github.com/amoghmunikote/cmpunlocker"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3.json","alternate":"/r/qwen3-8-flash-next-w4a16-int4ple-on-2x-modded-cmp-170hx-pp2-mtp3"}},{"slug":"qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g","title":"Qwen3.8-Flash-Next W4A16-AutoRound DGX Spark vLLM MTP4 Iter7g","summary":"Recipe by kyaky. Qwen3.8-Flash-Next W4A16-AutoRound hybrid (INT4 experts + FP8 side layers + INT8 head; 51B-token n-gram PLE table on NVMe mmap) on one DGX Spark (GB10, 121 GB unified memory) on a verified-deterministic vLLM floor (image qwen38-flash-dgx:iter7g-20261015 on the digest-pinned vllm/vllm-openai:qwen38-flash-next base): MoE align arrival-order shim, drafter-Marlin atomicAdd fix, pinned triton config, compile-cache mount. Measured line v19a-w5 = MTP=4 window-5 block verification: 76.8 tok/s single-stream, 261.2 tok/s aggregate at 8 streams (+9.2% N1 / +4.8% N8 over MTP=3, two boots per arm, same image). Promoted serving line v20-fp8: fp8 KV cache on a 24 GB pool at native 262,144 context, 1,291,120-token capacity. Chinese +31.3% via a 25,862-id (39.5%) Han draft-vocab rebalance (prose-zhen 34.8 to 45.7 tok/s). Stochastic-correctness gate 37/37 vs bit-exact fp64 oracle; per-round raw data + scripts ship in-repo; copy-agentic-1 failure disclosed.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":61.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:28.107205-06","commit_sha":"dc8982acabf5e6b6417f4871ad5149d61f01755d","content_hash":"e6a2cfdf4903c7ca1b80df3d0f496d3ca3f13840","votes":0,"credit":[{"name":"kyaky","url":"https://github.com/kyaky/qwen3.8-flash-next-gb10-ultrafast"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g.json","alternate":"/r/qwen3-8-flash-next-w4a16-autoround-dgx-spark-vllm-mtp4-iter7g"}},{"slug":"qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp","title":"Qwen3.8-Flash-Next NVFP4 DGX Spark GB10 SGLang NEXTN MTP","summary":"Production single-node DGX Spark (Grace Blackwell GB10, SM121, 128GB unified memory) recipe for Qwen3.8-Flash-Next on SGLang: ModelOpt NVFP4 quantization with zero-copy NVMe mmap of the 47.7 GiB ple_table.bin via --ple-offload-embedding (eliminates 51 GiB of unified-memory footprint), custom Triton FP4 prefill kernels + trtllm_mha decode kernels on SM121, NEXTN MTP (3 verification steps / 4 draft tokens) lifting output to 110-152 tok/s, GDN+QSA hybrid (131,072-token served window, 262k native capacity), 96-slot mamba recurrence buffers, sharp froggeric v22.1 Jinja chat template (-18% response length), persistent Triton/Inductor SM121 JIT caches on NVMe, 8-worker parallel NVMe pre-warm (206 shards in 17s) + 16-core Grace OMP. Author-measured vs same-box 27B dense baseline: 110.4-152.8 tok/s decode w/ MTP (baseline 12.1 base / 21.5 MTP), single-stream CoT 25.1-28.8 tok/s (2.62x speculative gain), TTFT ~347ms radix hit / ~465ms cold, 113 GiB / 121 GiB RAM used (~8 GiB headroom), 103.5 GiB","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":110.400000000000005684341886080801486968994140625,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:27.869274-06","commit_sha":"2c5220aa85b5379d7373abd0aa14bb13b1ec993f","content_hash":"bf0c02f95b99494d6720a515e2f8f438f126bc87","votes":0,"credit":[{"name":"Michael Bemler (Bemler Labs)","url":"https://github.com/bemlerlabs/qwen3.8-flash-next-dgx-spark-sglang"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp.json","alternate":"/r/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-nextn-mtp"}},{"slug":"qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6","title":"Qwen3.8-Flash-Next NVFP4 2x CMP 170HX TEP2 GDS SSD PLE MTP6","summary":"Recipe by nguyenthimy2022kg-alt. Community patched vLLM run of the NVIDIA Qwen3.8-Flash-Next NVFP4 checkpoint on 2x unlocked CMP 170HX (SM80, ~63.4 GiB usable each) with TEP2 tensor+expert parallel and fixed MTP6 speculative decoding with per-request draft isolation. v0.3.0 supports 1-10 active requests (max_num_seqs accepts 1-10, defaults to 1 when omitted; repo example sets 10) with compact/topk vocabulary transport and expanded PCIe communication; history-copy (ngram hybrid) drafting removed. The ~51.2 GB FP8 PLE table stays on NVMe, read GPU-direct via GDS/cuFile with 32 persistent readers and 3 staging slots - no CPU-memory relay; optional host-read PLE backend off by default. FlashInfer PCIe IPC AllReduce, 36-token prefix matching, bayley/cmpunlocker BAR1/P2P driver + host GDS. Measured upstream (2026-10-11, thinking off): 10 requests x 5,000 tokens, 749.67 tok/s aggregate, 76.29/request; historical 2026-09-08 window 151.05-168.29 tok/s decode at 8K-128K inputs xhigh thinking.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":158.19999999999998863131622783839702606201171875,"prefill_tok_s":2059.90000000000009094947017729282379150390625,"updated_at":"2026-10-29 21:32:27.639988-06","commit_sha":"7d911c7dd2ce6b0a45d3c14791ee74036a43e7ae","content_hash":"247ee93d0bf55ac542edef7091844ac3df34a323","votes":0,"credit":[{"name":"nguyenthimy2022kg-alt","url":"https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6.json","alternate":"/r/qwen3-8-flash-next-nvfp4-2x-cmp-170hx-tep2-gds-ssd-ple-mtp6"}},{"slug":"qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x","title":"Qwen3.8-Flash-Next FP4-Local 1x DGX Spark vLLM b12x","summary":"Recipe by sojufx. Serves NVIDIA's nvidia/Qwen3.8-Flash-Next-FP4-Local (96 GiB NVFP4 quantization-aware-distilled checkpoint in a new compressed NVFP4 scale format) on one DGX Spark (GB10) via the digest-pinned eugr/spark-vllm-b12x community container (reported vLLM 0.1.dev21830+g8244d1c44.d20261015) - the model card requires community b12x containers until the format lands upstream. FP8 KV cache, exact MTP K=3 with standard rejection sampling, 262,144-token context, 4 scheduler slots, 4096-token batched prefill, disk-backed io_uring PLE, prefix cache with Mamba align, vision plus qwen3_xml tool calling, thinking off by default, localhost-only 127.0.0.1:8001 OpenAI-compatible backend with a deliberate no-replace/no-port-take guard. Measured 2026-10-15/16 on the author's Spark: 66.2 tok/s single-stream sparkDash counting decode, 166.4 tok/s at C4, 41-54 tok/s single-stream on the four-prompt wall-clock suite - different measurement bases, not interchangeable.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":66.2000000000000028421709430404007434844970703125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:27.177524-06","commit_sha":"5fc0d101d8e9b1c6a1419f3d465e21649f4ecbad","content_hash":"be42677d0cc5ca7f71fe72210e756f1622fe3604","votes":0,"credit":[{"name":"sojufx","url":"https://github.com/sojufx/Qwen3.8-Flash-Next-FP4-Local-DGX-Spark-Recipe"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x.json","alternate":"/r/qwen3-8-flash-next-fp4-local-1x-dgx-spark-vllm-b12x"}},{"slug":"qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm","title":"Qwen3.8-Flash-Next 3bpw FP8-PLE 1x CMP 170HX DFlash2 RecoverSSM","summary":"Qwen3.8-Flash-Next (125B MoE) 3 bpw AutoRound experts on ONE CMP 170HX card with the full 262,144-token context: 197.8/296.4/135.5 tok/s one user (structured/code/prose), 371 tok/s editing code, 799.5/791.3/586.4 across eight, up to 48 requests at once - 1,384 tok/s total with the default DFlash2 drafter (1,403 with the checkpoint's MTP head), 4,243 tok/s cold prefill, 292,784-token KV pool. Serves klee100's 3 bpw AutoRound checkpoint (3-bit g128 / 2-bit g64 experts, 47 GiB on card) with halt95's FP8 n-gram table in HOST RAM (48 GiB, hard-linked at build) on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) so the whole decode step is one CUDA graph on a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3 by request count) + the recipe's RecoverSSM + BF16 GDN-state patches (9 files mounted over the image) fit 48 concurrent requests on one card. Measured 2026-10-23 under the MiaAI-Lab protocol.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":197.80000000000001136868377216160297393798828125,"prefill_tok_s":0,"updated_at":"2026-10-29 21:32:26.955646-06","commit_sha":"af7d7dde12376551476c8b9e36cd72c68dc32408","content_hash":"c82e80ffa9955d905b61bda381cf7d89db1350ba","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-3bpw-1x-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"klee100 (3 bpw AutoRound checkpoint with its MTP head, made for a 64 GiB card; not involved in this recipe)","url":"https://huggingface.co/klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP"},{"name":"halt95 (W4A16-Merlin checkpoint whose FP8 n-gram table this recipe serves; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound quantisation) + RadixArk (FP8 n-gram table quantisation)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/Mia-AiLab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm.json","alternate":"/r/qwen3-8-flash-next-3bpw-fp8-ple-1x-cmp-170hx-dflash2-recoverssm"}},{"slug":"qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple","title":"Qwen3.8-Flash-Next W4A16 2x CMP 170HX TP2 DFlash2 host-RAM PLE","summary":"Qwen3.8-Flash-Next (125B MoE) INT4-expert W4A16 on two CMP 170HX cards, full 262,144-token context: 241/350/198 tok/s one user (structured/code/prose), 495 tok/s editing code, 895/987/770 across eight, up to 128 requests at once totaling 1,937 tok/s (DFlash 2; 1,966 MTP), 4,696 tok/s cold prefill, 1,475,836-token KV pool. Serves halt95's W4A16-Merlin checkpoint on Morrowmake's vllm-cmp170hx fork @ 18305948acba (stock release image, digest-pinned) with the 51 GB FP8 n-gram table in HOST RAM (cpu_offload; PLE_SHARED_DIR shares one copy across servers), the whole decode step one CUDA graph - built for a 2016 dual-Xeon host. inco.ai's DFlash2 drafter (7/5/3/2 schedule, 44% faster than MTP on code c1, 83% editing code) via engine-file patches plus RecoverSSM one-state GDN; optional KV offload restores an evicted 100k-token context in 0.89 s (21 s without). Measured 2026-10-18/23 from a fresh clone under Morrowmake's protocol (temp 0, 400-token pin, median-of-5 c1 / median-of-3 aggregates).","tier":"community","beta":false,"generation":"ampere","decode_tok_s":253.19999999999998863131622783839702606201171875,"prefill_tok_s":0,"updated_at":"2026-10-29 20:26:39.946771-06","commit_sha":"d558ee307e9364af70f6a3c8317aa1db958be441","content_hash":"da435f81c05c69636771a5f32003ae1c413e0bf2","votes":0,"credit":[{"name":"ElectroHobo","url":"https://github.com/ElectroHobo/qwen38-flash-next-w4a16-cmp170hx-recipe"},{"name":"Morrowmake (vllm-cmp170hx engine + image + Qwen EXL3 recipe scripts this descends from, MIT)","url":"https://github.com/Morrowmake/vllm-cmp170hx"},{"name":"halt95 (W4A16-Merlin checkpoint + companion 3090 engine; not involved in this recipe)","url":"https://huggingface.co/halt95/Qwen3.8-Flash-Next-W4A16-Merlin"},{"name":"inco.ai (DFlash2 drafter, Apache-2.0)","url":"https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2"},{"name":"Intel (AutoRound INT4 experts) + RadixArk (FP8 n-gram table) + DominikBucko (MTP-head packing)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"},{"name":"Qwen (Qwen3.8-Flash-Next base model, Qwen Community License 1.0)","url":"https://huggingface.co/Qwen/Qwen3.8-Flash-Next"},{"name":"MiaAI-Lab (benchmark protocol + category prompts, used as data)","url":"https://huggingface.co/MiaAI-Lab"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple.json","alternate":"/r/qwen3-8-flash-next-w4a16-2x-cmp-170hx-tp2-dflash2-host-ram-ple"}},{"slug":"qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce","title":"Qwen3.8-Flash-Next Uncensored NVFP4 2x DGX Spark SGLang TP2 ROCE","summary":"Recipe by ttlogan. Production 2-node GB10 (ASUS GX10 / DGX Spark) cluster serve of jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4 (OrcaRouter uncensored weights repacked to ModelOpt NVFP4 with the PLE n-gram table in FP8, ~126 GiB) on the MiaAI native vLLM stack --tp 2 2-node over ConnectX/ROCE (slug name says SGLang: historical identifier - the repo now runs vLLM at ec4d38624b), MTP speculative decoding, and a REQUIRED qwen4_exp tokenizer registry overlay (BPE tokenizer.json, NOT stock tokenizer behaviour). Podman rootless deployment with boot autostart and a deepseek/glm/qwen swap orchestrator. Measured numbers unchanged (no re-measurement in d5bd003301..ec4d38624b): 40.3 tok/s single, 117.3 tok/s agg at c=4 (the locked optimum), prefill ~2900-3000 tok/s flat; 30-min c=4 soak clean (2432 req / 0 fail / health 200). License notes: PixelML patches MIT; QSA fp8-KV overlay vendored AGPL-3.0-or-later from the single-Spark recipe.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":40.2999999999999971578290569595992565155029296875,"prefill_tok_s":2936.6999999999998181010596454143524169921875,"updated_at":"2026-10-28 16:04:52.791091-06","commit_sha":"1217f615379960f758d79b04ceb4ca2a71e8e6cc","content_hash":"db96361cfc2e1999cd1d8ce8f34678e090df5f9b","votes":0,"credit":[{"name":"ttlogan","url":"https://github.com/ttlogan/qwen38-flash-next-recipe"},{"name":"PixelML/MiaAI (SM121 QSA fallback + token-0 guard patches, MIT)"},{"name":"jpezzulli (OrcaRouter uncensored ModelOpt NVFP4 + FP8 PLE repack)","url":"https://huggingface.co/jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce.json","alternate":"/r/qwen3-8-flash-next-uncensored-nvfp4-2x-dgx-spark-sglang-tp2-roce"}},{"slug":"qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data","title":"Qwen3.8-Flash-Next NVFP4 1x DGX Spark vLLM 71.3 c1 (salient-data)","summary":"Qwen3.8-Flash-Next (~180B multimodal MoE, NVFP4) on one NVIDIA DGX Spark (GB10, 121 GiB unified memory) with a pinned vLLM nightly image + 14 provenance-tracked patches (patch --fuzz=0) + one docker run, fitting the 51.2 GB PLE n-gram table by never making it resident (rows read per forward-pass step from the checkpoint file). 69.3 tok/s single-stream median decode (runs 66.8/69.5/69.3), 41.0 prose median, 91.9 tok/s 6-stream aggregate (1 run), GSM8K 99/100 on a fixed 100-problem subset (temp 0.6 top-p 0.95 thinking on), 12/12 arithmetic + 131K needle pass + prefix-cache canary pass, prefix caching ON. Measured with tonyd2wild's bench_categories.py @ 6ad1c8f unchanged on build 2026-10-22-8e2beaa3; clean first-time run 2026-10-25 re-measured 69.8/41.5/82.6. Prod/dev: prod image digest-pinned + cosign-signed + SPDX SBOM; dev rebuilds the same serving code from published patches with an AST-equivalence proof. Outside the model card's support statement (B200/B300 listed, GB10 unlisted).","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":69.2999999999999971578290569595992565155029296875,"prefill_tok_s":0,"updated_at":"2026-10-28 15:59:39.152527-06","commit_sha":"85e2e4120ef69829bff82736c39745ac579c9a6b","content_hash":"63f86642d07e8833929005e2087044adaecc1ca4","votes":0,"credit":[{"name":"Jeremy Newhouse / salient-data (recipe, patches, measurements)","url":"https://github.com/salient-data/qwen38-next-dgx-recipe"},{"name":"tonyd2wild (bench_categories.py @ 6ad1c8f + comparator stack/numbers)","url":"https://github.com/tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark"},{"name":"NVIDIA (Qwen3.8-Flash-Next-NVFP4 checkpoint; HF token gated)","url":"https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4"},{"name":"vLLM project (nightly base image 8e2beaa3)","url":"https://github.com/vllm-project/vllm"},{"name":"Sigstore/cosign (image signing + SBOM timestamping)","url":"https://docs.sigstore.dev"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data.json","alternate":"/r/qwen3-8-flash-next-nvfp4-1x-dgx-spark-vllm-71-3-c1-salient-data"}},{"slug":"qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes","title":"Qwen3.8-Flash-Next W4A16 2x CMP 170HX PP2 MTP5 PLE (AlexBrandes)","summary":"Recipe by AlexBrandes. Qwen3.8-Flash-Next (Intel/Qwen3.8-Flash-Next-W4A16-AutoRound @ 4c67bf68) on two NVIDIA CMP 170HX cards in PIPELINE parallel (PP2, layer split 26/22) with a vLLM v0.30.0 fork (AlexBrandes/vllm @ 02acf67c, 6 commits) in a digest-pinned container. Native MTP with 5 draft tokens (lm_head quantized to int4 by tools/quantize_lm_head.py, sha256 pinned), reasoning + tool calls, full 262,144-token window. The ~102 GB n-gram (PLE) table stays on NVMe read through mmap (VLLM_PLE_MMAP=1); BF16 KV, prefix caching, FP32 Mamba state. FIRST 2x170HX recipe with NO P2P or link modification needed - PP sends one activation per step, so it runs on stock PCIe Gen2 x4 links (all sibling rows need the unlock/P2P-mod prerequisite). Measured 2026-10-17 with sparkDash (3e6d4ef): 226 tok/s single-user code, 840 tok/s 8 users, 5,885-7,104 tok/s prefill 4k-128k; HumanEval 157/164, GSM8K 1,271/1,319, needle 12/12.","tier":"community","beta":false,"generation":"ampere","decode_tok_s":226.19999999999998863131622783839702606201171875,"prefill_tok_s":0,"updated_at":"2026-10-27 20:30:16.824026-06","commit_sha":"44b6d37a739c1ee1ea463ffb4775b87382ef4c39","content_hash":"035586de62fecf1d4dec99de529bd4f53e8a3aa5","votes":0,"credit":[{"name":"AlexBrandes (recipe, fork, image, measurements)","url":"https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next"},{"name":"AlexBrandes (vllm fork: PP + NVMe-mmapped PLE, Mamba PP state, draft lm_head, graph-safe PLE, bounded prefill)","url":"https://github.com/AlexBrandes/vllm/tree/cmp170hx-qwen38-flash-next"},{"name":"polyuij42-del (community CMP 170HX W4A16 work the fork builds on)","url":"https://github.com/polyuij42-del/qwen38-flash-next-w4a16-cmp170hx"},{"name":"nguyenthimy2022kg-alt (community SM80 170HX work the fork builds on)","url":"https://github.com/nguyenthimy2022kg-alt/Qwen-Flash-SM80-170HX"},{"name":"MiaAI-Lab (sparkDash benchmark protocol @ 3e6d4ef)","url":"https://github.com/MiaAI-Lab/sparkDash"},{"name":"Intel (INT4 W4A16 AutoRound quant)","url":"https://huggingface.co/Intel/Qwen3.8-Flash-Next-W4A16-AutoRound"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes.json","alternate":"/r/qwen3-8-flash-next-w4a16-2x-cmp-170hx-pp2-mtp5-ple-alexbrandes"}}]}