NVIDIA DEVELOPER FORUMS·vLLM·Hybrid FP8 / NVFP4 / MXFP4

GLM-5.2 Hybrid FP8 / NVFP4 / MXFP4

Per-component mixed-precision GLM-5.2 — FP8 attention, NVFP4 dense/shared experts, MXFP4 routed experts — running across a 4-node DGX Spark GB10 cluster.

4× DGX Spark GB10 (121GB each) · Hybrid FP8 / NVFP4 / MXFP4 · vLLM · source NVIDIA Developer Forums

GLM-5.2
vLLM
multi-node
DGX Spark
mixed precision
quantization
MoE
long context

4× DGX Spark GB10Hardware

FP8 / NVFP4 / MXFP4Precision

vLLM (custom b12x fork)Runtime

262,144 tokensMax context

01Overview

Different precision per component — FP8 attention, NVFP4 dense/shared experts, MXFP4 routed experts — so the model fits across 4 nodes.
Ships as a ready-to-run Docker image with custom vLLM kernels precompiled; no manual build.
Known bug: the image bakes in an env var that breaks GLM-5.2's KV coordinator. The launcher unsets it automatically.
Early community release. The author notes the kernel code has not been cleaned up for public release.

The condensed setup

Download the checkpoint: hf download aidendle94/GLM-5.2-Hybrid-FP8-MXFP4 --local-dir /models/glm52-hybrid
Pull the image on all 4 nodes: docker pull aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.2
Confirm fast interconnect (InfiniBand or RoCE) between all 4 nodes, and note your NIC name / IB HCA.
Run the container on each node (recipe below) — RANK=0 on the head node, RANK=1,2,3 on the three workers.
First boot takes ~15–20 min (weight load + JIT kernel compile). Later boots reuse the JIT cache (~6 min).
Check readiness: curl http://<head>:8210/health
Query the OpenAI-compatible endpoint at /v1/chat/completions.

02The recipe

Docker launch (run on every node)

docker run -d --name glm52-hybrid \
  --network host --ipc host --shm-size 10gb --gpus all \
  --cap-add IPC_LOCK --ulimit memlock=-1:-1 \
  --device /dev/infiniband:/dev/infiniband \
  -v /models/glm52-hybrid:/hybridmodel:ro \
  -v $HOME/glm-jit-cache:/cache/jit \
  -e VLLM_USE_V2_MODEL_RUNNER=1 -e VLLM_USE_B12X_MOE=1 \
  -e VLLM_USE_B12X_SPARSE_INDEXER=1 \
  -e VLLM_DCP_SHARD_DRAFT=1 -e VLLM_DCP_GLOBAL_TOPK=1 \
  -e VLLM_DSV4_INDEXER_SP=1 \
  -e VLLM_B12X_MLA_CKV_GATHER=1 \
  -e VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS=262144 \
  -e VLLM_B12X_MLA_DECODE_SPARSE_GATHER=1 \
  -e VLLM_B12X_MLA_DECODE_GATHER_V2=1 \
  -e VLLM_ADAPTIVE_SPEC_DEPTHS=2,4 \
  -e VLLM_NVFP4_MLA_PER_TOKEN_SCALE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e CUTE_DSL_ARCH=sm_121a -e TORCH_CUDA_ARCH_LIST=12.1a \
  -e XDG_CACHE_HOME=/cache/jit \
  -e NCCL_NET=IB -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_HCA=$HCA -e NCCL_IB_GID_INDEX=3 \
  -e NCCL_SOCKET_IFNAME=$NIC -e GLOO_SOCKET_IFNAME=$NIC \
  -e NCCL_MAX_NCHANNELS=4 -e NCCL_MIN_NCHANNELS=4 \
  -e NCCL_CUMEM_ENABLE=0 \
  -e RANK=$RANK -e HEAD_IP=$HEAD_IP \
  aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.2 \
  bash -lc 'unset VLLM_PREFIX_CACHE_RETENTION_INTERVAL; exec vllm serve /hybridmodel \
    --served-model-name glm-5.2 --host 0.0.0.0 --port 8210 \
    --trust-remote-code --reasoning-parser glm45 \
    --tool-call-parser glm47 --enable-auto-tool-choice \
    --enable-prefix-caching \
    --tensor-parallel-size 4 --decode-context-parallel-size 4 \
    --dcp-comm-backend ag_rs \
    --attention-backend B12X_MLA_SPARSE \
    --hf-overrides "{\"index_topk_pattern\":\"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS\"}" \
    --kv-cache-dtype nvfp4_ds_mla --kv-cache-memory-bytes 5500000000 \
    --max-model-len 262144 --max-num-batched-tokens 2048 \
    --max-num-seqs 3 \
    --gpu-memory-utilization 0.89 \
    --speculative-config "{\"model\":\"/hybridmodel/mtp-draft\",\"method\":\"mtp\",\"num_speculative_tokens\":4,\"draft_attention_backend\":\"B12X_MLA_SPARSE\",\"adaptive_speculative_tokens_window\":32}" \
    --compilation-config "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"custom_ops\":[\"all\"],\"cudagraph_capture_sizes\":[1,2,3,4,5,6,8]}" \
    --distributed-timeout-seconds 3600 \
    --cpu-distributed-timeout-seconds 3600 \
    --distributed-executor-backend mp \
    --nnodes 4 --node-rank $RANK --master-addr $HEAD_IP \
    --master-port 29501'

03Technical details

GLM-5.2 is too large to run at usable context lengths on 4 unified-memory nodes at a single precision. This recipe assigns precision per component instead: attention at FP8, the routed MoE experts at MXFP4, and dense/shared components at NVFP4 as a middle ground.

What is quantized

Attention layers: Block-FP8 (128×128, compressed-tensors format), from a Red Hat quantization.
Dense MLPs & shared experts: NVFP4 (group-16), also from Red Hat.
Routed experts (layers 3–77): MXFP4 (e2m1 + ue8m0, per-32 groups), taken unmodified from an AMD/community ("festr2") calibration and stitched in without re-quantizing.
Norms, gates, lm_head: left in BF16.
KV cache: a custom NVFP4 format (nvfp4_ds_mla) for full capacity; stock vLLM falls back to fp8 KV cache at roughly half the capacity.

The three quantized sources are assembled as a "byte-exact transplant" — tensors copied in from each source unmodified, unified with a hand-authored model.safetensors.index.json rather than re-running quantization end-to-end.

Runtime settings

Per-token KV scaling cuts quantization error (KL-divergence 0.40 vs a BF16 reference).
Adaptive speculative-decoding depths: pinning depth 4 caused a decode-throughput regression the adaptive setting avoids.
Batching and GPU-memory settings validated at roughly 812 tok/s prefill with a KV pool usable up to ~875K tokens.

Known issue: a baked-in env var

The shipped image bakes in VLLM_PREFIX_CACHE_RETENTION_INTERVAL, but GLM-5.2 doesn't do sliding-window KV grouping, so the KV coordinator rejects the value at startup. The launch command unsets the variable right before invoking vllm serve. One community member solved it differently with a derived Dockerfile and a wrapper entrypoint (see reported results below).

bash

curl http://<head>:8210/health

curl http://<head>:8210/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"glm-5.2","messages":[{"role":"user","content":"Capital of Estonia?"}],"max_tokens":64}'

This is an early, community-maintained release. The author's model card notes that the kernel code has not been cleaned up for public release and requests feedback.

04Reported results

Encountered the same baked-in VLLM_PREFIX_CACHE_RETENTION_INTERVAL issue on a 4×Spark setup and worked around it with a derived Dockerfile: a small entrypoint wrapper unsets the variable before handing off to the original NVIDIA entrypoint (exec /opt/nvidia/nvidia_entrypoint.sh "$@").
— Alexander Korolev · source
Responded positively to the 25 tok/s prose throughput figure in the original recipe; a reaction, not an independent re-measurement.
— CosmicRaisins · source

original thread / model card →