{"meta":{"count":2,"page":1,"pages":1,"per_page":12,"applied":{"engine":null,"model":null,"quant":null,"storage":null,"hardware":["dgx-spark-x3","rtx-5090-x2"],"generation":null,"task":null,"tier":null,"tags":[],"vram":null,"search":null,"sort":"updated","featured":false},"origin":"https://localairecipes.com","openapi":"/openapi.json","updated_at":"2026-10-31T13:40:07+00:00"},"recipes":[{"slug":"glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw","title":"GLM-5.3-Flash EXL3 2.05bpw on 2x RTX 5090 - TabbyAPI hot-expert swaps, 262K ctx, vision","summary":"GLM-5.3-Flash (321B MoE, ~18B active) turboderp EXL3 2.05bpw on two RTX 5090 + 64 GB DDR5: TabbyAPI on ExLlamaV3 with CPU-offloaded routed experts (96/layer GPU0 layers 3-22+MTP, 102/layer GPU1 23-44), runtime hot-expert swaps, per-GPU expert split, the DSA indexer's per-token rows in a fixed ring, live /metrics token counters, the expert-exchange clock handed to the first text MoE layer so swaps also run at 2-4 streams, MTP head drafting 1 tok/step single-stream, 262,144-token window on an 8-bit KV page pool (~1.6 GiB across 4), vision ON with tower linears pinned in host RAM. Exact-output: no expert skipped, every draft verified. Measured 2026-10-10 (R969, three fresh boots): c1 score 62.9 tok/s (code 59.9/prose 66.7/chat 68.5/html 64.8/edit 53.9), 2/3/4-stream sums 74.4/92.0/98.0. Nine in-repo overlays on the qwen3.8-flash-next-2x-rtx5090 base image.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":62.89999999999999857891452847979962825775146484375,"prefill_tok_s":2023,"updated_at":"2026-10-29 23:29:37.491895-06","commit_sha":"4dc491797af361a6a0f4cdedde5f7dbeb918082d","content_hash":"66ee933dc865f3362d1e2880b2ef8e330f331b05","votes":0,"credit":[{"name":"adrienbrault (recipe, five patch overlays, measurements)","url":"https://github.com/adrienbrault/glm5.3-flash-2x-rtx5090"},{"name":"zai-org (GLM-5.3-Flash)","url":"https://huggingface.co/zai-org/GLM-5.3-Flash"},{"name":"turboderp (EXL3 2.05bpw quantisation; ExLlamaV3 + CPU MoE offload)","url":"https://huggingface.co/turboderp/GLM-5.3-Flash-exl3"},{"name":"theroyallab (TabbyAPI)","url":"https://github.com/theroyallab/tabbyAPI"},{"name":"adrienbrault/qwen3.8-flash-next-2x-rtx5090 (base image: ExLlamaV3 dev 5783a93 patch stack + TabbyAPI 53da7919)","url":"https://github.com/adrienbrault/qwen3.8-flash-next-2x-rtx5090"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw.json","alternate":"/r/glm-5-3-flash-exl3-2-05bpw-on-2x-rtx-5090-tabbyapi-hot-expert-sw"}},{"slug":"glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4","title":"GLM-5.3-Flash exlr8 EXL3 experts on DGX Spark TP1-4","summary":"GLM-5.3-Flash exlr8: MoE layers are EXL3-trellis units decoded by the custom exlr8 MoE kernel; dense linears stay FP8/NVFP4. One release carries three expert widths (k2/k3/k4 = 2/3/4 bpw), chosen at boot via VLLM_EXLR8_WIDTHS. Reference topology TP=4 on four GB10 boxes; TP=1 serves the whole model (~87 GiB) on one 128 GB DGX Spark / ASUS GX10 via tp1.yaml plus a distinct MENTAT_GROUP. TP=2 and TP=3 also measured. 2026-10-19, clocks TP-locked at 2200 MHz: decode code 128.0 tok/s, prose 72.5 at TP=4; 53.0 / 29.5 at TP=1; prefill 4,393 tok/s at 8K, 4,190 at 32K. Trade vs the shipping NVFP4 build: decode +3-8%, NLL equal or better, KV pool ~+15%, prefill 7.7% lower at Second 1x-Spark GLM-5.3-Flash lane: not the VQ Q4_K GGUF lane (/r/glm-5-3-flash-vq-q4-k-1x-dgx-spark-1m-ctx-jlfn), and not a fork of the 2x ATLAS build (/r/glm-5-3-flash-nvfp4-on-2x-dgx-spark-atlas-rc4-2-tp2-dflash2-1m) whose NVFP4 Flash base this stack overlays.","tier":"community","beta":false,"generation":"blackwell","decode_tok_s":65.2000000000000028421709430404007434844970703125,"prefill_tok_s":3438.5,"updated_at":"2026-10-29 21:32:25.385656-06","commit_sha":"2ddd4d836d8efa003f7203ba30597c20dc21e495","content_hash":"79db03010b10026e9788b7c749df25daf5fb38be","votes":0,"credit":[{"name":"kindlingai","url":"https://github.com/kindlingai/glm-5.3-flash-exlr8-gx10"},{"name":"nvidia","url":"https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4"}],"uploaded_by":"localai-recipes","measured":null,"links":{"self":"/api/v1/recipes/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4.json","alternate":"/r/glm-5-3-flash-exlr8-exl3-experts-on-dgx-spark-tp1-4"}}]}