unsloth/GLM-5.3-Flash-FP8 — 328 GB
- vLLM-native: no tensor remapping, MTP head present,
marlinis a valid FP8 MoE backend. The configuration carries over unchanged.
GLM-5.3-Flash on four DGX Spark GB10 nodes: the SM90 NoPE MLA attention route, DFlash2 speculation, CUDA graphs and a host memory floor. ~90 tok/s decode, 393K context.
4× DGX Spark GB10 · TP4/EP4 · vLLM 1.2 · aidendle94/sparkrun-vllm-glm53-gb10:production-1.2 · measured Aug 31, 2026
GLM-5.3-Flash served across four DGX Spark GB10 nodes (TP4 over dual 200G RoCE). Builds on tonyd2wild's NVFP4 1M-KV recipe for the attention route and FujitsuPolycom's TP4 Spark recipe for the NCCL ring profile.
GB10 is unified memory, and its NVRM allocator uses __GFP_RETRY_MAYFAIL: it requires actual MemFree and will not evict page cache. A node showing 45 GiB of cache and 2 GiB free fails a GPU allocation that a node with 45 GiB genuinely free serves. drop_caches moves 1–4 GiB free to 45–51 GiB immediately.
--kv-cache-memory-bytes instead of letting the profiler size it — at FP8, weights are ~82 GB/rank and MemFree sits at 5–9 GB, the edge of the failure zone./etc/sysctl.d/: vm.min_free_kbytes=1048576, vm.dirty_bytes=536870912, vm.dirty_background_bytes=134217728, vm.dirty_expire_centisecs=500, vm.dirty_writeback_centisecs=100, vm.swappiness=10. Dirty pages are the one kind of cache drop_caches cannot free; capping them at 512 MiB and keeping 1 GiB genuinely free stops a download, log burst or checkpoint copy from starving the next GPU allocation. Do not go to 2 GiB: the higher watermark lowers MemAvailable by the same amount and puts a 121 GB node onto earlyoom's 2% line.GPU KV cache size line rather than scaling the token count by the ratio — that arithmetic does not hold on this model (see the context section).marlin is a valid FP8 MoE backend. The configuration carries over unchanged.One container per node, host networking, RDMA passthrough. The fabric variables are the only site-specific values. NCCL_IB_HCA must list both RoCE devices: each Spark has two active 200G links, and pinning NCCL to one is pinning it to half the fabric.
docker run -d --name glm53flash \
--network host --ipc host --shm-size 32g --gpus all \
--cap-add IPC_LOCK --ulimit memlock=-1:-1 \
--memory 112g --memory-swap 112g \
--device /dev/infiniband:/dev/infiniband \
-v $HOME/.cache/huggingface:/cache/huggingface \
-e HF_HOME=/cache/huggingface \
# --- your fabric (per cluster / per node) ---
-e NCCL_IB_HCA=<dev0>,<dev1> # e.g. rocep1s0f0,roceP2p1s0f0 — list both
-e NCCL_SOCKET_IFNAME=<your-fabric-iface> \
-e GLOO_SOCKET_IFNAME=<your-fabric-iface> \
-e VLLM_HOST_IP=<this-node-fabric-ip> \
# --- address the fabric by subnet, not by GID index ---
-e NCCL_IB_ROCE_VERSION_NUM=2 \
-e NCCL_IB_ADDR_FAMILY=AF_INET \
-e NCCL_IB_ADDR_RANGE=10.0.0.0/24 \
# --- ring profile (FujitsuPolycom) ---
-e NCCL_NET=IB -e NCCL_IB_DISABLE=0 \
-e NCCL_ALGO=Ring -e NCCL_PROTO=LL,LL128,Simple \
-e NCCL_SKIP_TREE_CONNECT=1 \
-e NCCL_IB_SUBNET_AWARE_ROUTING=1 -e NCCL_IB_SUBNET_PREFIX_LEN=24 \
-e NCCL_IB_MERGE_NICS=0 -e NCCL_P2P_LEVEL=SYS -e NCCL_NET_PLUGIN=none \
-e NCCL_MAX_NCHANNELS=4 -e NCCL_MIN_NCHANNELS=4 \
-e NCCL_CROSS_NIC=1 -e NCCL_CUMEM_ENABLE=0 -e NCCL_IGNORE_CPU_AFFINITY=1 \
# --- housekeeping ---
-e TORCH_USE_RTLD_GLOBAL=1 -e TORCHINDUCTOR_COMPILE_THREADS=1 \
-e CUTE_DSL_ARCH=sm_121a -e TORCH_CUDA_ARCH_LIST=12.1a \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
aidendle94/sparkrun-vllm-glm53-gb10:production-1.2 \
bash -lc 'unset VLLM_PREFIX_CACHE_RETENTION_INTERVAL; exec <the serve command below>'Start workers (ranks 1..N-1) headless first, then rank 0. Before any boot, run sync && echo 3 > /proc/sys/vm/drop_caches on every node — see the memory wall.
vllm serve <path-to>/GLM-5.3-Flash-FP8 \
--served-model-name <your-alias> --host 0.0.0.0 --port 8210 \
--trust-remote-code \
--tensor-parallel-size 4 \
--nnodes 4 --node-rank $RANK --master-addr $HEAD_IP --master-port 29513 \
--distributed-executor-backend mp --disable-custom-all-reduce \
--distributed-timeout-seconds 240 --cpu-distributed-timeout-seconds 3600 \
# functional parity — each of these fixes a silent client-facing gap
--reasoning-parser glm45 \
--tool-call-parser glm47 --enable-auto-tool-choice \
--chat-template <path-to>/chat_template_mm.jinja \
--default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "high"}' \
# the SM90-NoPE + FA2 route: plain fp8 cache, real qk_rope_head_dim=0
--kv-cache-dtype fp8_e4m3 \
--block-size 2304 \
--moe-backend marlin \
--max-model-len 393216 --max-num-batched-tokens 8192 --max-num-seqs 16 \
--enable-chunked-prefill \
--gpu-memory-utilization 0.85 \
--kv-cache-memory-bytes 12884901888 \
--speculative-config '{"method":"dflash","model":"<path-to>/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7}'--enforce-eager is gone in 1.2. CUDA graphs measured neutral on 1.0/1.1, but that reading was taken at the 16 GiB pin, where graphs pushed the head node into the earlyoom zone during long prefill; at 12 GiB they are worth +4–19% decode by regime and cost under 1 GB and ~30 s of capture. Prefill is unaffected either way.--gpu-memory-utilization 0.85 is not arbitrary: 0.78 and 0.80 both starve the KV cache.production-1.2 (2026-09-04): 1.1 plus CUDA graphs, the 12 GiB KV pin and a fused hyper-connection (mHC) Triton kernel on the prefill path. Same weights, same drafter, same flags otherwise; greedy answers agree with 1.1 on 30/30 keyed prompts.production-1.1 is the eager predecessor: the SM90-NoPE-to-capability-12 attention route, the three functional fixes, and the DFlash2 overlay (drafter package, aux-capture patch, KV-page LCM patch, backend-registry patch).production-1.0 is the same stack without DFlash2. It runs the checkpoint's native MTP head out of the box: use '{"method":"mtp","num_speculative_tokens":5}' and expect ~66 tok/s structured instead of 84.GLM5Next is NoPE (qk_rope_head_dim=0) — 45 layers: 34 KDA linear attention, 11 sparse MLA (kv_lora_rank=512, 64 heads, index_topk=2048, index_kpool=4). Two routes exist on SM121.
pe_dim==64: rope padding via --hf-overrides, per-head q_b_proj padding for TP4 sharding, and custom kernels for the 2176-wide buffer.bf16/fp8_e4m3 cache and a real qk_rope_head_dim=0.fp8_ds_mla, the q_b_proj padding and the 2176 kernels. This is what the published image ships.Without speculation this stack runs 20.6 tok/s, flat across output regimes. DFlash2 (incoai/GLM-5.3-Flash-DFlash2) is the larger win: 110 vs 66 tok/s structured on the NVFP4 build, 5.06 accepted tokens per step vs the native MTP head's 2.81.
The chat template opens with {%- set effective_reasoning_effort = reasoning_effort if reasoning_effort is defined and reasoning_effort in ['low', 'high'] else 'max' -%}. Three consequences:
max works by accident; medium silently becomes Max.'{"enable_thinking": false}' configuration renders a contradictory instruction on every request: reason at maximum effort, then a pre-closed empty thinking block forbidding it. The closed block wins, so behaviour is correct — but it is still contradictory.| effort | reasoning produced |
|---|---|
low | 0 words — thinking is on, and the model closes it immediately |
high | ~36 words |
max (or unset) | ~420 words |
One puzzle prompt, three samples each. Max is more than 10× High, and scales with difficulty: at High, an easy factual question drew ~3 reasoning words while the puzzle drew ~36–47.
The shipped default is '{"enable_thinking": true, "reasoning_effort": "high"}' — stated explicitly rather than relying on the fall-through. Cost on short prompts is +0.5 s end-to-end (1.75 → 2.27 s). Raw decode is largely unaffected (84.1 → 89.3 tok/s structured), except highly predictable output like a repeated alphabet, which drops from 82.5 to 55.2 tok/s because reasoning prose is less draftable. Clients can still override per request with reasoning_effort: "low" or enable_thinking: false.
The checkpoint's native limit is 1,048,576 tokens (text_config.max_position_embeddings), so anything up to 1M is inside the trained range — no rope scaling, no VLLM_ALLOW_LONG_MAX_MODEL_LEN fiction. A modest --max-model-len here is your own conservatism, not a model limit.
Going from 131,072 to 393,216 cost nothing measurable in decode: 83.6 / 83.0 / 76.3 / 28.8 tok/s after the change against 84.8 / 82.7 / 75.0 / 28.8 before. Per-token work does not grow with context, so neither does the cost of *allowing* more of it.
| `--max-model-len` | KV pool | full-length concurrency |
|---|---|---|
| 131,072 | 1,581,771 tokens | 12.07× |
| 393,216 | 2,252,384 tokens | 5.73× |
With --kv-cache-memory-bytes pinned at exactly 16 GiB across both runs (the 1.1 pin; at the 1.2 pin of 12 GiB the 393,216 row is 1,688,835 tokens, 4.29×). The pool grew 42% from a change that should only have moved a ceiling: on this hybrid KDA-plus-MLA model the page geometry is negotiated against the mamba state, and that negotiation lands differently. Read your own GPU KV cache size line rather than predicting it.
DCP shards each layer's KV across ranks and roughly doubles the pool at DCP2; on the DeepSeek-V4 vision stack it does exactly that. On GLM-5.3-Flash it is not worth doing: only 11 of 45 layers hold real KV (the 34 KDA layers keep a small per-request recurrent state), which is why 2.25M tokens already fit with no sharding. What blocks it is the KpoolTailSpec — the one-block-per-request circular tail that feeds the KV-pool compressor — which is neither a full-attention, Mamba, nor replicated group, so the hybrid KV coordinator's DCP branch asserts. Making it replicated is the trick that made DSpark run under DCP on DeepSeek, but the GLM image sits on an older vLLM tree with none of the replicated-group plumbing, and its kpool compress and DFlash2 draft paths have no DCP awareness. Several days of porting for 2× on a pool that is not the constraint.
Measured 2026-09-04. Four-node TP4 · FP8 · marlin MoE · DFlash2 k=7 · CUDA graphs · 12 GiB KV pin per rank · thinking on at High by default. Decode is a streamed 400-token completion per regime with completion_tokens accounting; prefill is unique-content cold prefill with a needle check. Image: aidendle94/sparkrun-vllm-glm53-gb10:production-1.2.
Same harness, same nodes, same day. CUDA graphs take launch overhead out of the decode step; the gain is largest where acceptance is lowest.
The shipped default. Reasoning prose is less draftable than the answer that follows it, so alphabet-style output no longer collapses the way it did on 1.1.
Within run-to-run noise (±3%). Prefill at TP4 is bound by attention, the tensor-parallel all-reduce and the MoE; graphs do not touch it.
GB10 is unified memory and earlyoom fires at 2% MemAvailable (~2.4 GB). The 16 GiB configurations touched that line during long prefill; at 12 GiB the floor through a 128K prefill is 6.4 GB.
−25% pool for ~4 GB per rank of host headroom. 4.29 full-length contexts still fit, and only 11 of 45 layers hold real KV on this model.
| production-1.1 | production-1.2 | |
|---|---|---|
| CUDA graphs | off (--enforce-eager); graphs had measured neutral | on — +4–19% decode by regime (structured 86.4 → 89.7, code 75.3 → 80.7, freeform 28.5 → 33.8 tok/s). The earlier "neutral" reading was taken at the 16 GiB pin, where graphs pushed the head node into the earlyoom zone during long prefill |
--kv-cache-memory-bytes | 16 GiB per rank — KV pool 2,252,384 tokens (5.73×) | 12 GiB per rank — 1,688,835 tokens (4.29×); head-node MemAvailable 2.4–4.7 → 6.4–7.5 GB |
| Hyper-connection (mHC) kernel | three TileLang kernels per layer | one fused Triton kernel for prefill tiles of 1,024+ tokens: 1.49× on that section in isolation, +1–2% end-to-end prefill; VLLM_MHC_FUSED_LARGE=0 disables it |
| Prefill | 2,073–2,124 tok/s at 32K–128K | 2,045–2,150 tok/s — unchanged within noise |
| Quality | keyed 30/30 | keyed 30/30, greedy outputs agree with 1.1 on 30/30 |
| Tried and rejected | — | deep_gemm MoE backend (−12% prefill, −19% decode); Triton FP8 W8A8 block-scaled MoE (4.7% relative error vs 0.6% for marlin); 16K prefill chunks (head-node OOM at the 16 GiB pin, untested at 12 GiB) |
| Weights, drafter, flags | unsloth FP8, DFlash2 k=7, thinking on at High | unchanged |
| Host memory floor (2026-09-10) | kernel defaults: 45 MB minimum free, dirty pages up to 20% of RAM | 1 GiB minimum free, dirty pages capped at 512 MiB with 1 s writeback, swappiness 10 — after a download on the head killed the engine with cudaErrorNotPermitted on 2026-09-09 |
| Concurrency validation (2026-09-10) | single-stream benches only | 4 and 8 concurrent pp2048/tg128 (2,192 and 2,263 tok/s aggregate prefill) under a 12 GB page-cache write, 3×64K coherence soak, decode 88.5/88.4/79.3/33.2, quality 30/30, 32K/128K prefill 2,090/2,071 — CUDA graphs on |
Same weights and nodes on both sides unless the row says otherwise.
Different prompt sets, all single stream at temperature 0 with completion_tokens accounting: exact within a recipe, indicative across them. Inkling-2-Small · SGLang (experimental): code questions of 400 tokens (60–61), DSpark with the confidence gate, experts sharded by width; 56.6 at temperature 0.7 · DeepSeek-V4.1-Flash · SGLang (1.8): code prompts, DSpark block 5, RoCE collectives, early Engram staging, V4.1-Flash-1017; 113.7 counting, 34.9 prose · DeepSeek-V4.1-Flash (1.1, DCP2): code prompts, DSpark k=5, b12x MoE, DCP2 (77.5 on 1.0 with a different prompt set) · GLM-5.3-Flash (1.2): structured output, thinking off · DeepSeek-V4-Flash (3.76.1): mixed prompts, 8-rep mean · DeepSeek-V4-Flash-Vision (3.73): code prompts, DCP2 + DSpark
Cold means the prompt has never been seen by the server; with prefix caching on, a repeated prompt is a cache hit and reads 3–25× higher. Inkling-2-Small · SGLang (experimental): 130K depth, cold (2,417–2,482 at ~15K) · DeepSeek-V4.1-Flash · SGLang (1.8): 32K depth, unique random words (3,848 at 128K) · DeepSeek-V4.1-Flash (1.0): 36K depth (1,940 at 132K) · GLM-5.3-Flash (1.2): 32K depth · DeepSeek-V4-Flash (3.76.1): 334K depth · DeepSeek-V4-Flash-Vision (3.73): 92K depth (2,088 at 334K)
Decode is flat with context: 91 → 96 tok/s from 805 to 93,534 tokens. 34 of 45 layers are KDA linear attention (O(1) in context); the 11 MLA layers are sparse at index_topk=2048, so per-token attention work stops growing past ~2K. Long contexts are cheap to serve and expensive to prefill.
| Missing flag | Symptom |
|---|---|
--reasoning-parser glm45 | Chain-of-thought leaks into message.content instead of the reasoning field |
--tool-call-parser glm47 --enable-auto-tool-choice | Requests with tool_choice: auto return HTTP 400 |
--chat-template <mm template> | Image requests return 500 — the checkpoint ships a text-only template |
The server comes up healthy without these.
One of the four units reset silently three times, each time one to three minutes into a sustained all-node long prefill, with nothing in the journal. Sampling every node every two seconds under identical load showed that unit's GPU 10 °C hotter and drawing ~25% more power than its twin, on identical firmware, with the board's critical trip at 104.8 °C. Capping that one GPU at 90% of its application clock (nvidia-smi -lgc 0,2176, persisted with a oneshot systemd unit) took it from 87–89 °C to 69 °C under the same load for a 1.2% cluster prefill cost. It has not reset since. Stream the sensor samples to another machine: a log written on the node that resets dies with it.
cudaErrorNotPermitted — which is how a checkpoint download on the head took the engine down seven seconds after it started. Six sysctls (memory-wall section) close it./health, never /v1/models./v1/models returns 200 with a dead engine; /health returns 503 on EngineDeadError.docker rm -f destroys the evidence. NVRM failures surface in logs minutes after the event.completion_tokens, not SSE chunks.Speculative decoding emits multiple tokens per chunk; chunk counting made a 3× speedup read as a 30% regression.persistent_topk engages./metrics settles it: unchanged acceptance with lower throughput means the speculator is fine and the GPU is shared.docker run to a healthy /health on 2026-09-03. Size watchdog ready-timeouts accordingly; the fleet watchdog here uses 3600 s.NCCL_IB_GID_INDEX.RoCE GID tables renumber when NIC addresses change — a switch outage moved the entry from index 3 to 4 and every collective failed with ibv_modify_qp … ENODATA. Subnet addressing (NCCL_IB_ADDR_RANGE et al.) is immune.