NVIDIA Developer Forums

GLM-5.2

A mixed-precision build of GLM-5.2 — FP8 attention, NVFP4 dense/shared experts, MXFP4 routed experts — tuned to run across a 4-node DGX Spark GB10 cluster with an 800K+ token context window.

GLM-5.2
vLLM
multi-node
DGX Spark
mixed precision
quantization
MoE
long context

Hardware4× DGX Spark GB10

PrecisionFP8 / NVFP4 / MXFP4

RuntimevLLM (custom b12x fork)

Max context262,144 tokens

Different precision per component — FP8 attention, NVFP4 dense/shared experts, MXFP4 routed experts — so the model fits across 4 memory-constrained nodes.
Ships as a ready-to-run Docker image with custom vLLM kernels precompiled — no manual build.
Known bug: the image bakes in an env var that breaks GLM-5.2's KV coordinator. The recipe below unsets it automatically.
Early, community-maintained release; the author notes the kernel code has not yet been prepared for public release.

Download the checkpoint: hf download aidendle94/GLM-5.2-Hybrid-FP8-MXFP4 --local-dir /models/glm52-hybrid
Pull the image on all 4 nodes: docker pull aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.2
Confirm fast interconnect (InfiniBand or RoCE) between all 4 nodes, and note your NIC name / IB HCA.
Run the container on each node (recipe below) — RANK=0 on the head node, RANK=1,2,3 on the three workers.
First boot takes ~15–20 min (weight load + JIT kernel compile). Later boots reuse the JIT cache (~6 min).
Check readiness: curl http://<head>:8210/health
Query the OpenAI-compatible endpoint at /v1/chat/completions.

The recipe

Docker launch (run on every node)

docker run -d --name glm52-hybrid \
  --network host --ipc host --shm-size 10gb --gpus all \
  --cap-add IPC_LOCK --ulimit memlock=-1:-1 \
  --device /dev/infiniband:/dev/infiniband \
  -v /models/glm52-hybrid:/hybridmodel:ro \
  -v $HOME/glm-jit-cache:/cache/jit \
  -e VLLM_USE_V2_MODEL_RUNNER=1 -e VLLM_USE_B12X_MOE=1 \
  -e VLLM_USE_B12X_SPARSE_INDEXER=1 \
  -e VLLM_DCP_SHARD_DRAFT=1 -e VLLM_DCP_GLOBAL_TOPK=1 \
  -e VLLM_DSV4_INDEXER_SP=1 \
  -e VLLM_B12X_MLA_CKV_GATHER=1 \
  -e VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS=262144 \
  -e VLLM_B12X_MLA_DECODE_SPARSE_GATHER=1 \
  -e VLLM_B12X_MLA_DECODE_GATHER_V2=1 \
  -e VLLM_ADAPTIVE_SPEC_DEPTHS=2,4 \
  -e VLLM_NVFP4_MLA_PER_TOKEN_SCALE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e CUTE_DSL_ARCH=sm_121a -e TORCH_CUDA_ARCH_LIST=12.1a \
  -e XDG_CACHE_HOME=/cache/jit \
  -e NCCL_NET=IB -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_HCA=$HCA -e NCCL_IB_GID_INDEX=3 \
  -e NCCL_SOCKET_IFNAME=$NIC -e GLOO_SOCKET_IFNAME=$NIC \
  -e NCCL_MAX_NCHANNELS=4 -e NCCL_MIN_NCHANNELS=4 \
  -e NCCL_CUMEM_ENABLE=0 \
  -e RANK=$RANK -e HEAD_IP=$HEAD_IP \
  aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.2 \
  bash -lc 'unset VLLM_PREFIX_CACHE_RETENTION_INTERVAL; exec vllm serve /hybridmodel \
    --served-model-name glm-5.2 --host 0.0.0.0 --port 8210 \
    --trust-remote-code --reasoning-parser glm45 \
    --tool-call-parser glm47 --enable-auto-tool-choice \
    --enable-prefix-caching \
    --tensor-parallel-size 4 --decode-context-parallel-size 4 \
    --dcp-comm-backend ag_rs \
    --attention-backend B12X_MLA_SPARSE \
    --hf-overrides "{\"index_topk_pattern\":\"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS\"}" \
    --kv-cache-dtype nvfp4_ds_mla --kv-cache-memory-bytes 5500000000 \
    --max-model-len 262144 --max-num-batched-tokens 2048 \
    --max-num-seqs 3 \
    --gpu-memory-utilization 0.89 \
    --speculative-config "{\"model\":\"/hybridmodel/mtp-draft\",\"method\":\"mtp\",\"num_speculative_tokens\":4,\"draft_attention_backend\":\"B12X_MLA_SPARSE\",\"adaptive_speculative_tokens_window\":32}" \
    --compilation-config "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"custom_ops\":[\"all\"],\"cudagraph_capture_sizes\":[1,2,3,4,5,6,8]}" \
    --distributed-timeout-seconds 3600 \
    --cpu-distributed-timeout-seconds 3600 \
    --distributed-executor-backend mp \
    --nnodes 4 --node-rank $RANK --master-addr $HEAD_IP \
    --master-port 29501'

Reported results

Encountered the same baked-in VLLM_PREFIX_CACHE_RETENTION_INTERVAL issue while bringing the recipe up on a 4×Spark setup, and worked around it differently than the recipe above: a derived Dockerfile that layers a small entrypoint wrapper to unset the variable before handing off to the original NVIDIA entrypoint (exec /opt/nvidia/nvidia_entrypoint.sh "$@").
— Alexander Korolev · source
Responded positively to the 25 tok/s prose throughput figure reported in the original recipe; a reaction to the reported number, not an independently re-measured benchmark.
— CosmicRaisins · source

original thread / model card →