GLM-5.3-Flash TP4
A 45-layer hybrid of KDA linear attention and NoPE sparse MLA, served TP4 across four DGX Spark GB10 nodes at ~90 tok/s over a 393,216-token context.
4× DGX Spark GB10 (121GB each) · FP8 e4m3 (or NVFP4) · vLLM · source tonyd2wild / GLM-5.3-Flash-NVFP4-1M-KV-4x-DGX-Spark
- GLM-5.3-Flash
- vLLM
- multi-node
- TP4
- DGX Spark
- FP8
- MoE
- linear attention
01Overview
- GLM-5.3-Flash is NoPE (`qk_rope_head_dim=0`); the SM120 sparse-MLA path hardcodes a positional dimension of 64, so it needs faking the model shape and then wedges the GPU.
- The SM90 NoPE backend extended to capability 12 with FA2 accepts the real shape and removes the rope padding, the fp8_ds_mla cache and the custom kernel instantiations.
- Decode is flat with context — 91 to 96 tok/s from 800 to 93K tokens — because 34 of 45 layers are linear attention. Prefill is where long context costs you.
- A DFlash2 block-diffusion drafter at k=7 is worth roughly 65% decode over no speculation, but only single-stream; under concurrency it matches the native MTP head.
- The checkpoint is natively 1,048,576 tokens, so a 393,216-token context needs no rope scaling.
- Set reasoning_effort explicitly: the chat template honours only 'low' and 'high', and silently treats anything else — including unset — as Max (~10× High).
The condensed setup
- Fetch the weights: hf download unsloth/GLM-5.3-Flash-FP8 (328 GB), or LibertAIDAI/GLM-5.3-Flash-NVFP4 (194 GB) for ~23% more throughput; plus the incoai/GLM-5.3-Flash-DFlash2 drafter (2.34 GB) as $DRAFT_PATH.
- Pull the image on all 4 nodes: docker pull aidendle94/sparkrun-vllm-glm53-gb10:production-1.2 — CUDA graphs, DFlash2 and the 12 GiB KV pin out of the box.
- Flush page cache on every node first: sync && echo 3 > /proc/sys/vm/drop_caches — GB10 needs real MemFree and will not evict cache to get it.
- Start ranks 1-3 headless, then rank 0. First boot ~10-15 min; later boots reuse the Triton cache.
- Check readiness with /health, never /v1/models — the latter returns 200 with a dead engine.
02The recipe
docker run -d --name glm53flash \
--network host --ipc host --shm-size 32g --gpus all \
--cap-add IPC_LOCK --ulimit memlock=-1:-1 \
--memory 112g --memory-swap 112g \
--device /dev/infiniband:/dev/infiniband \
-v $HOME/.cache/huggingface:/cache/huggingface \
-e HF_HOME=/cache/huggingface \
-e NCCL_IB_HCA=$HCA0,$HCA1 \
-e NCCL_SOCKET_IFNAME=$NIC -e GLOO_SOCKET_IFNAME=$NIC \
-e VLLM_HOST_IP=$NODE_IP \
-e NCCL_IB_ROCE_VERSION_NUM=2 -e NCCL_IB_ADDR_FAMILY=AF_INET \
-e NCCL_IB_ADDR_RANGE=10.0.0.0/24 \
-e NCCL_NET=IB -e NCCL_IB_DISABLE=0 \
-e NCCL_ALGO=Ring -e NCCL_PROTO=LL,LL128,Simple \
-e NCCL_SKIP_TREE_CONNECT=1 \
-e NCCL_IB_SUBNET_AWARE_ROUTING=1 -e NCCL_IB_SUBNET_PREFIX_LEN=24 \
-e NCCL_IB_MERGE_NICS=0 -e NCCL_P2P_LEVEL=SYS -e NCCL_NET_PLUGIN=none \
-e NCCL_MAX_NCHANNELS=4 -e NCCL_MIN_NCHANNELS=4 \
-e NCCL_CROSS_NIC=1 -e NCCL_CUMEM_ENABLE=0 -e NCCL_IGNORE_CPU_AFFINITY=1 \
-e TORCH_USE_RTLD_GLOBAL=1 -e TORCHINDUCTOR_COMPILE_THREADS=1 \
-e CUTE_DSL_ARCH=sm_121a -e TORCH_CUDA_ARCH_LIST=12.1a \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
aidendle94/sparkrun-vllm-glm53-gb10:production-1.2 \
bash -lc 'unset VLLM_PREFIX_CACHE_RETENTION_INTERVAL; exec vllm serve $MODEL_PATH \
--served-model-name glm-5.3-flash --host 0.0.0.0 --port 8210 \
--trust-remote-code \
--reasoning-parser glm45 \
--tool-call-parser glm47 --enable-auto-tool-choice \
--chat-template $TEMPLATE_PATH \
--default-chat-template-kwargs "{\"enable_thinking\": true, \"reasoning_effort\": \"high\"}" \
--kv-cache-dtype fp8_e4m3 \
--block-size 2304 \
--moe-backend marlin \
--tensor-parallel-size 4 \
--max-model-len 393216 --max-num-batched-tokens 8192 --max-num-seqs 16 \
--enable-chunked-prefill \
--gpu-memory-utilization 0.85 \
--kv-cache-memory-bytes 12884901888 \
--speculative-config "{\"method\":\"dflash\",\"model\":\"$DRAFT_PATH\",\"num_speculative_tokens\":7}" \
--distributed-timeout-seconds 240 \
--cpu-distributed-timeout-seconds 3600 \
--disable-custom-all-reduce \
--distributed-executor-backend mp \
--nnodes 4 --node-rank $RANK --master-addr $HEAD_IP \
--master-port 29513'03Technical details
GLM-5.3-Flash is 45 layers: 34 KDA linear-attention layers and 11 NoPE sparse-MLA layers with kv_lora_rank=512, 64 heads, index_topk=2048 and index_kpool=4. Per-token attention work stops growing once context passes roughly 2K, so decode speed is essentially independent of context length while prefill remains linear.
Why the obvious backend is the wrong one
The FLASHINFER_MLA_SPARSE_SM120 path with an fp8_ds_mla cache hardcodes a positional-embedding dimension of 64. Because this model has none, using that path means faking pe_dim==64: padding the rope dimension through --hf-overrides, padding q_b_proj per-head so TP4 sharding does not overflow its slice, and compiling custom kernel instantiations for a 2176-wide index buffer. It compiles and loads, then wedges the GPU.
The working route is the SM90 NoPE MLA backend extended to compute capability 12, with FA2 selected off-Hopper. It accepts a plain bf16 or fp8_e4m3 cache and a genuine qk_rope_head_dim of 0, and every piece of scaffolding above becomes unnecessary.
Three gaps that stay silent until real traffic arrives
- Without --reasoning-parser glm45, chain-of-thought leaks into message.content instead of the reasoning field.
- Without --tool-call-parser glm47 --enable-auto-tool-choice, any request carrying tool_choice: auto returns HTTP 400.
- Without a multimodal chat template, image requests return 500 — the checkpoint ships a text-only template.
The GB10 memory wall
GB10 is a unified-memory part, and its NVRM allocator uses __GFP_RETRY_MAYFAIL: it requires genuine MemFree and will not evict page cache to obtain it. A node showing 45 GiB of cache and 2 GiB free fails allocations that an otherwise identical node with 45 GiB actually free serves without complaint. Flush caches before every boot, run a low-rate flusher for the life of the deployment, and pin the KV slab with --kv-cache-memory-bytes instead of letting the profiler size it.
Reasoning effort is a trap worth reading twice
The chat template begins: reasoning_effort if reasoning_effort in ['low','high'] else 'max'. Only low and high are honoured; every other value falls through to Max, so 'medium' silently becomes Max — and leaving the field unset means Max rather than none. That system line is emitted regardless of enable_thinking, so the common enable_thinking:false configuration ships a contradictory instruction: reason at maximum effort, then a pre-closed empty think block forbidding it. The closed block wins, but the instruction is contradictory. Measured on one puzzle prompt: low produced 0 reasoning words, high ~36, max ~420.
Speculative decoding
Without speculation this stack sits at 20.6 tok/s, flat across every output regime — acceptance-driven throughput should vary with how predictable the output is. The DFlash2 block-diffusion drafter requires num_speculative_tokens = block_size - 1 = 7. Acceptance should land between 0.5 and 0.8; ~0.15 indicates a broken auxiliary capture or mHC contraction, which degrades silently. The native MTP head is the alternative and peaks at k=5, not the k=4 most launchers ship.
curl http://<head>:8210/health
# acceptance and concurrency in one call — if acceptance is unchanged
# but throughput is down, you are sharing the GPU, not regressing
curl -s http://<head>:8210/metrics | grep -E "num_requests_running|spec_decode_num_(accepted|draft)_tokens_total"04Reported results
No community results reported yet — run it and be the first to share what you get.