tonyd2wild / GLM-5.3-Flash-NVFP4-1M-KV-4x-DGX-Spark

GLM-5.3-Flash

A 45-layer hybrid of KDA linear attention and NoPE sparse MLA, served TP4 across four DGX Spark GB10 nodes at ~84 tok/s over a 393,216-token context, with decode throughput that does not degrade as context grows.

GLM-5.3-Flash
vLLM
multi-node
TP4
DGX Spark
FP8
MoE
linear attention
speculative decoding
flat decode

Hardware4× DGX Spark GB10

PrecisionFP8 e4m3 / NVFP4

RuntimevLLM (SM90-NoPE + FA2 route)

Max context393,216 tokens

The whole problem is backend selection: GLM-5.3-Flash is NoPE (qk_rope_head_dim=0), and the SM120 sparse-MLA path hardcodes a positional dimension of 64.
Taking the SM90 NoPE backend extended to capability 12 with FA2 removes the rope padding, the fp8_ds_mla cache and the custom kernel instantiations all at once — none of them were solving a real problem.
Decode is flat with context — 91 to 96 tok/s from 800 to 93K tokens — because 34 of 45 layers are linear attention and the rest are sparse. Prefill is where long context costs you.
A DFlash2 block-diffusion drafter at k=7 is worth roughly 65% decode over no speculation, but only single-stream; under concurrency it measures identical to the native MTP head.
The checkpoint is natively 1,048,576 tokens, so a 393,216-token context needs no rope scaling — and raising the ceiling from 131,072 cost nothing measurable in decode throughput.
Set reasoning_effort explicitly: the chat template honours only 'low' and 'high', and silently treats anything else — including unset — as Max, which produces roughly 10x the reasoning of High.

Fetch the weights: hf download unsloth/GLM-5.3-Flash-FP8 (328 GB), or LibertAIDAI/GLM-5.3-Flash-NVFP4 (194 GB) for ~23% more throughput.
Pull the image on all 4 nodes: docker pull aidendle94/sparkrun-vllm-glm53-gb10:production-1.0
Flush page cache on every node first: sync && echo 3 > /proc/sys/vm/drop_caches — GB10 needs real MemFree and will not evict cache to get it.
Start ranks 1-3 headless, then rank 0. First boot ~10-15 min; later boots reuse the Triton cache.
Check readiness with /health, never /v1/models — the latter returns 200 with a dead engine.

The recipe

Docker launch (workers 1-3 headless first, then rank 0)

docker run -d --name glm53flash \
  --network host --ipc host --shm-size 32g --gpus all \
  --cap-add IPC_LOCK --ulimit memlock=-1:-1 \
  --memory 112g --memory-swap 112g \
  --device /dev/infiniband:/dev/infiniband \
  -v $HOME/.cache/huggingface:/cache/huggingface \
  -e HF_HOME=/cache/huggingface \
  -e NCCL_IB_HCA=$HCA0,$HCA1 \
  -e NCCL_SOCKET_IFNAME=$NIC -e GLOO_SOCKET_IFNAME=$NIC \
  -e VLLM_HOST_IP=$NODE_IP \
  -e NCCL_IB_ROCE_VERSION_NUM=2 -e NCCL_IB_ADDR_FAMILY=AF_INET \
  -e NCCL_IB_ADDR_RANGE=10.0.0.0/24 \
  -e NCCL_NET=IB -e NCCL_IB_DISABLE=0 \
  -e NCCL_ALGO=Ring -e NCCL_PROTO=LL,LL128,Simple \
  -e NCCL_SKIP_TREE_CONNECT=1 \
  -e NCCL_IB_SUBNET_AWARE_ROUTING=1 -e NCCL_IB_SUBNET_PREFIX_LEN=24 \
  -e NCCL_IB_MERGE_NICS=0 -e NCCL_P2P_LEVEL=SYS -e NCCL_NET_PLUGIN=none \
  -e NCCL_MAX_NCHANNELS=4 -e NCCL_MIN_NCHANNELS=4 \
  -e NCCL_CROSS_NIC=1 -e NCCL_CUMEM_ENABLE=0 -e NCCL_IGNORE_CPU_AFFINITY=1 \
  -e TORCH_USE_RTLD_GLOBAL=1 -e TORCHINDUCTOR_COMPILE_THREADS=1 \
  -e CUTE_DSL_ARCH=sm_121a -e TORCH_CUDA_ARCH_LIST=12.1a \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  aidendle94/sparkrun-vllm-glm53-gb10:production-1.0 \
  bash -lc 'unset VLLM_PREFIX_CACHE_RETENTION_INTERVAL; exec vllm serve $MODEL_PATH \
    --served-model-name glm-5.3-flash --host 0.0.0.0 --port 8210 \
    --trust-remote-code \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 --enable-auto-tool-choice \
    --chat-template $TEMPLATE_PATH \
    --default-chat-template-kwargs "{\"enable_thinking\": true, \"reasoning_effort\": \"high\"}" \
    --kv-cache-dtype fp8_e4m3 \
    --block-size 2304 \
    --moe-backend marlin \
    --tensor-parallel-size 4 \
    --max-model-len 393216 --max-num-batched-tokens 8192 --max-num-seqs 16 \
    --enable-chunked-prefill \
    --enforce-eager \
    --gpu-memory-utilization 0.85 \
    --kv-cache-memory-bytes 17179869184 \
    --speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":5}" \
    --distributed-timeout-seconds 240 \
    --cpu-distributed-timeout-seconds 3600 \
    --disable-custom-all-reduce \
    --distributed-executor-backend mp \
    --nnodes 4 --node-rank $RANK --master-addr $HEAD_IP \
    --master-port 29513'

Reported results

No community results reported yet — run it and be the first to share what you get.

original thread / model card →