DeepSeek-V4.1-Flash-1017 SAGE EXL3 1.62bpw 1x DGX Spark DSpark

'bash packs/1.62bpw/scripts/download_and_verify.sh (hf_xet download of vcruz305/DSV4.1-Flash-1017-SAGE-EXL3-1.62bpw @ c852150e06f5... + sha256 verify vs packs/1.62bpw/scripts/SHA256SUMS - the SUMS file is the authority, the pinned revision may stop resolving); bash scripts/build_image.sh; bash packs/1.62bpw/scripts/build_cold_alias.sh (8.59 GiB read-only side file, ~10-20 s)' 'bash packs/1.62bpw/scripts/serve.sh: host wrapper - sets vm.compaction_proactiveness=0 vm.watermark_boost_factor=0 vm.swappiness=0, swapoff/swapon, drop_caches, refuses a stale alias manifest (serve.sh compares ALIAS_DIR/manifest.json against configs/cold-alias-selection.json) or an image whose build labels do not match the pins (PLUGIN_REF 30e8b941376c, FORK_REF 909ad8564fcb, MOE_TILESIZE_K=16), then runs the docker command below and arms scripts/guard.sh. DRY_RUN=1 prints the plan.' 'Env knobs of the measured run (scripts/env.sh defaults): CONTAINER=dsv41-tp1 API_PORT=8000 SERVED_MODEL_NAME=dsv41 MAXLEN=524288 KVB=1073741824 MNBT=2112 LPT=2047 MTP_K=5 CG_MODE=FULL_DECODE_ONLY CAPTURE_SIZES=1,2,3,4,5,6,7 PREFIX_CACHING=1 DEFAULT_MAX_TOKENS=32768 DEFAULT_TEMPERATURE=0.6 DEFAULT_TOP_P=0.95 ALIAS_DIR=$HOME/.cache/dsv41-ds41f/alias-cold MOE_TILESIZE_K=16' 'sudo -n docker run -d --name dsv41-tp1 --gpus all --memory=118g --memory-swap=118g --oom-score-adj 1000 --ulimit core=0 --shm-size=2g --security-opt=seccomp=unconfined --security-opt=label=disable --ulimit=memlock=-1:-1 -v $HOME/models:/models:ro -v $HOME/.cache/dsv41-ds41f:/cache:rw -v $HOME/.cache/dsv41-ds41f/alias-cold:/alias:ro -v <RUN>/ctr:/mem:rw -v packs/1.62bpw/scripts/serve_inner.sh:/root/serve.sh:ro -p 8000:8000 --entrypoint /bin/bash dsv41-1017-exl3:r6w /root/serve.sh' 'container env: HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 B12X_AUTOTUNE=0 B12X_BIND_COMPILE_WORKERS=2 B12X_STATE_COMPILE_WORKERS=2 B12X_WEIGHTS_COMPILE_WORKERS=4 B12X_DENSE_SPLITK_TURBO=1 B12X_W4A8_TINY_DECODE=0 CUDA_DEVICE_MAX_CONNECTIONS=32 VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_USE_FLASHINFER_SAMPLER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=900' 'container env (EXL3/ATS): VLLM_EXL3_MOE_KERNEL=exllamav3 VLLM_EXL3_V41_NATIVE_MOE=0 VLLM_EXL3_MIXEDK_LEGACY=1 VLLM_EXL3_MIXEDK_MODE=unified VLLM_EXL3_TRIM_AFTER_LAYER=1 VLLM_EXL3_MODEL_DIR=/models/dsv41-1.62bpw-v3 VLLM_EXL3_ATS_ALIAS=1 VLLM_EXL3_ATS_ALIAS_DIR=/alias VLLM_EXL3_ATS_ALIAS_LAYERS=38,39 VLLM_EXL3_ATS_ALIAS_MTP=1 VLLM_EXL3_ATS_ALIAS_EMBED=1 VLLM_EXL3_ATS_PREFETCH=1' 'container env (kernels/stops): VLLM_DS41_ROPE_TRIM=1 VLLM_DS41_ROPE_TRIM_MAX_POS=524288 VLLM_DS41_MARKOV_NVFP4=1 VLLM_DS41_L2_PREFETCH=0 VLLM_DS41_ATTENTION_COMPUTE=auto VLLM_DS41_KERNEL_BACKEND=tilelang VLLM_DS41_HEAD_DTYPE=float32 DSV41_DSML_STOP=1 DSV41_DSML_PARAM_REPAIR=1 DSV41_REP_STOP=64:512:8; JIT caches under $HOME/.cache/dsv41-ds41f/kkref/jit (mounted /cache)' 'scripts/guard.sh watch dsv41-tp1 <stage> <guard.log> 524288 <base_swap>: MUST be armed - 0.5 GiB MemAvailable floor, 64 MiB swap-growth allowance, 0.05 s sampling; kills the container on breach. guard.sh stop restores the sysctls' 'vllm serve /models/dsv41-1.62bpw-v3' --served-model-name dsv41 --host 0.0.0.0 --port 8000 --tokenizer-mode deepseek_v41 --dtype bfloat16 --tensor-parallel-size 1 --distributed-executor-backend uni --quantization exl3_dsv41 --block-size 256 --enable-prefix-caching --enable-chunked-prefill --max-model-len 524288 --max-num-seqs 1 --max-parallel-prefills 1 --max-num-batched-tokens 2112 --gpu-memory-utilization 0.90 --kv-cache-memory-bytes 1073741824 --generation-config vllm --override-generation-config '{"max_new_tokens": 32768,"temperature": 0.6,"top_p": 0.95}' --engram-config '{"table_memory":"disk","disk_resident_scales":false,"projection_tp":true}' --attention-backend tilelang --linear-backend tilelang --moe-backend tilelang --jit-monitor-mode warn --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","custom_ops":["all"],"cudagraph_capture_sizes":[1,2,3,4,5,6,7]}' --max-cudagraph-capture-size 7 --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_tensor_parallel_size":1,"attention_backend":"tilelang","draft_sample_method":"greedy","rejection_sample_method":"block","enable_adaptive_verification":true,"adaptive_verification_row_cost":0.01}' --reasoning-parser deepseek_v41 --tool-call-parser deepseek_v41 --enable-auto-tool-choice --long-prefill-token-threshold 2047 --async-scheduling --no-scheduler-reserve-full-isl --no-enable-flashinfer-autotune --language-model-only --mm-processor-cache-gb 0 --hf-overrides '{"quantization_config":{"quant_method":"exl3","bits":2,"codebook":"mcg","scope":"deepseek_v41_routed_experts","non_routed_quantization":{"quant_method":"deepseek_v41_fp8","weight_block_size":[32,32],"is_checkpoint_fp8_serialized":true,"activation_scheme":"dynamic"},"mtp_experts":"exl3"}}' 'Do NOT add --enforce-eager (the 74.4 ms/token no-draft path); do NOT raise --max-num-batched-tokens past 2112 (8192 OOMs); --gpu-memory-utilization is INERT because serve_inner.sh always passes --kv-cache-memory-bytes; MAXLEN=32768 is 1-5% faster on short prompts.' 'DEFAULT_MAX_TOKENS/DEFAULT_TEMPERATURE/DEFAULT_TOP_P (32768/0.6/0.95) exist because agent clients send none - without them replies ran 906 vs 277 completion tokens and 83% vs 89% valid tool calls. The repo also carries a sibling SAGE REXL3 1.55bpw pack (packs/1.55bpw-rexl3/, AgentSparko overlay, MAX_NUM_SEQS>1, vision) sharing the same engine - not part of this measured recipe. scripts/env.sh + packs/1.62bpw/scripts/env.sh are the pin authority.'

Fetch https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-sage-exl3-1-62bpw-1x-dgx-spark-dspark.json and apply the inference recipe it describes (engine flags, model, quant, context) to my setup. Ask before overwriting any existing config.

curl -fsSL https://localairecipes.com/api/v1/recipes/deepseek-v4-1-flash-1017-sage-exl3-1-62bpw-1x-dgx-spark-dspark.json