localairecipes launch configs for local inference

Qwen3.8-Flash-Next NVFP4 DGX Spark GB10 SGLang MTP-213 262K

0 sign in to vote

community Community submission — verify on your own hardware.

recipe by Felliks↗, sgl-project (SGLang + official MTP-213 profile)↗ · listed by localai-recipes

revision r2 · commit 28fe705dac · updated 10 d ago · license CC-BY-4.0

chat code mtp-speculative bf16-kv sglang nvme-ple dgx-spark gb10 sm121 felliks asus-ascent-gx10 eager-decode

Native-262K-context DGX Spark recipe for RadixArk/Qwen3.8-Flash-Next-NVFP4, validated on ASUS Ascent GX10 (same GB10 / 128GB unified memory as NVIDIA DGX Spark): the NVFP4 checkpoint is 135.24 GB and stock PLE host offload still lands the whole 51.2 GB table in host memory - but host memory and VRAM are the SAME 128 GB pool on GB10 - so this recipe adds two source-hash-guarded patches to a pinned SGLang image: (1) verified raw FP8 mmap keeps the full PLE table on NVMe with a bounded 256 MiB four-way pinned row cache for decode lookups (large prefill lookups bypass it), (2) QSA decode uses a PyTorch SDPA fallback on SM121 because the pinned FlashAttention-4 varlen path does not compile on GB10. prepare step converts the checkpoint's 128 FP8 PLE shards into a contiguous 51.2 GB mmap and byte-verifies 256 deterministic rows against the safetensors source. Official SGLang MTP-213 profile (2 speculative steps / top-k 1 / 3 draft tokens). Author-measured 2026-08-26 (benchmarks/asus-ascent-gx

Launch command

sglang serve --model-path RadixArk/Qwen3.8-Flash-Next-NVFP4 --revision 7b719225242aacd3dbd3f9407468c2ee9a9d2594 --served-model-name qwen38-flash-next-nvfp4-mtp --trust-remote-code --host 0.0.0.0 --port 30000 --quantization modelopt_fp4 --fp4-gemm-backend flashinfer_cutlass --page-size 64 --mamba-radix-cache-strategy extra_buffer --mamba-track-interval 64 --max-mamba-cache-size 20 --mamba-ssm-dtype float32 --chunked-prefill-size 4096 --max-running-requests 4 --max-total-tokens 524288 --context-length 262144 --mem-fraction-static 0.95 --allow-auto-truncate --ple-offload-embedding --reasoning-parser qwen3 --tool-call-parser qwen3_coder --preferred-sampling-params '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' --disable-prefill-cuda-graph --cuda-graph-backend-decode disabled --disable-flashinfer-autotune --speculative-algorithm NEXTN --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3

runtime

env: 'SGLANG_QWEN4_PLE_DISK_CACHE_PATH=/ple-cache/ple-fp8.raw | SGLANG_QWEN4_PLE_DISK_CACHE_MANIFEST=/ple-cache/ple-fp8.raw.json | SGLANG_QWEN4_PLE_SOURCE_REVISION=7b719225242aacd3dbd3f9407468c2ee9a9d2594 | SGLANG_QWEN4_PLE_CACHE_BYTES=268435456 | SGLANG_QWEN4_PLE_CACHE_MAX_LOOKUP_ROWS=4096'
notes: 'Serve profile = run-spark.sh serve() @ 5187a06c verbatim (docker run -d --rm --init --gpus all --ipc host --shm-size 16g --memory 116g --memory-swap 116g --cpuset-cpus 0-19 -p 127.0.0.1:8000:30000; mounts HF cache + sglang cache + ple-cache ro; local image qwen38-flash-next-gx10:73a255-ple-disk-v3). MTP-213 = SGLang official profile for this model (lmsys.org blog 2026-08-26): 2 steps / top-k 1 / 3 draft. KV BF16 (SM121 eager QSA path rejects FP8 K/V - tested). Decode EAGER by design: disk-fed PLE rows cannot be captured as fixed host pointers in a CUDA graph; prefill graph disabled. Default sampling for thinking = temp 1.0/top_p 0.95/top_k 20/min_p 0.0; non-thinking per request = temp 0.7/top_p 0.80/top_k 20/presence_penalty 1.5 (Qwen model card). Overrides: API_PORT, QWEN_NEXT_DATA_DIR, CPUSET, BIND_ADDR (default 127.0.0.1 - do not expose unauthenticated endpoint publicly). Known limits: 4 active requests share the 524K pool (two full 262K contexts max); TP=1 only for disk-backed PLE.'

Use this recipe with your agent

Paste this into any agent chat — it fetches and applies the recipe:

Fetch https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-mtp-213-262k.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

Or fetch it directly:

curl -fsSL https://localairecipes.com/api/v1/recipes/qwen3-8-flash-next-nvfp4-dgx-spark-gb10-sglang-mtp-213-262k.json

Engine

Install note: Fully pinned per repo 'Pinned components' table: base image lmsysorg/sglang @ sha256:14ed582518584c5c830206b5318a2c2769e68229c3422e48a28b952b3a888bd4 (image commit d91c3682b0b429e4c70df63cd57f819588ce29b0, qwen4 experimental source 73a255206f916366c8d26d4022f82ddfb0ab558d) + the repo's 2 source-hash-guarded patches (patch_qwen4_ple_disk_cache.py, patch_sm121_qsa_sdpa.py - the build FAILS if upstream files no longer match expected SHA-256) + prepare_ple.py mmap build w/ 256-row byte verification. Local image tag qwen38-flash-next-gx10:73a255-ple-disk-v3. Requires aarch64 + compute_cap 12.1 (run-spark.sh require_gb10 gate) and >=230GB free NVMe.

Model

Hardware

Context

  • max context: 256K tokens

KV pool

  • 512K tokens across concurrent requests

Performance

Speed by context

ContextConc.PrefillDecodeTTFT
262,1441—32.7 tok/s286 msreported
262,1444—93.2 tok/s—reported

prefill = prompt processing · decode = generation. All decode/prefill figures are shown as reported.

reported

notesAuthor-measured 2026-08-26 (benchmarks/asus-ascent-gx10-2026-08-26.json @ 5187a06c, structured record): single_stream 512 generated tokens, runs 28.078/32.772/32.727 tok/s, steady_state 32.7 tok/s, steady TTFT 0.2857s; four_stream_batch 4x768 tokens x2 reps wall 32.90/33.03s -> aggregate 93.372/92.996 tok/s, per-request range 23.636-23.807 tok/s, observed scheduler peak 103.36 tok/s; resident system memory 120.45 GB / 128 GB incl. OS; GPU utilization ~90%; cold load ~9.5 min. Configuration: ctx

Downloads

Revision notes

Ingested from GitHub repo Felliks/qwen38-flash-next-one-dgx-spark @ 5187a06c6c44a894ab6df2c7f07e005e1f8e66b9 (wave 3, kanban t_21a3a083). Validated on ASUS Ascent GX10; NVIDIA DGX Spark confirmations requested by author (same GB10 silicon - profile dgx-spark covers both, exact variant in tooltip/notes). Apache-2.0 code; Qwen Community License 1.0 weights. Measurements are from the included pinned checkpoint and launch path, not estimates copied from another GPU (author's statement).

Revision history

Fork

Sign in to fork this recipe — a fork is your own copy; you own it and only you can update it.

Export

recipe.yaml (raw)