AidenLabrecipe 02
MEASURED · 2026-11-15
MEASURED·12 min read·Aug 31, 2026

GLM-5.3-Flash on 4× DGX Spark

GLM-5.3-Flash on four DGX Spark GB10 nodes: the SM90 NoPE MLA attention route, DFlash2 speculation, CUDA graphs and a host memory floor. ~90 tok/s decode, 393K context.

4× DGX Spark GB10 · TP4/EP4 · vLLM 1.2 · aidendle94/sparkrun-vllm-glm53-gb10:production-1.2 · measured Aug 31, 2026

  • GLM-5.3-Flash
  • vLLM
  • multi-node
  • TP4
  • DGX Spark
  • FP8
  • MoE
  • linear attention
2,045–2,150tok/sPrefillcold, 32K–128K depth, needle found — unchanged from 1.1
88.5–89.7tok/sDecodesingle stream, structured output, thinking off
393KtokensContextverified with a 386,159-token prefill
1.69MtokensKV poolat 12 GiB/rank — 4.29 concurrent full-length contexts (2.25M at the 1.1 pin of 16 GiB)

01Overview

What you need
  • 4× DGX Spark GB10 nodes, fabric-linked (dual 200G RoCE)
  • aidendle94/sparkrun-vllm-glm53-gb10:production-1.2 image (Docker Hub) — DFlash2 drafter overlay, CUDA graphs, fused mHC kernel; 1.1 is the eager predecessor, 1.0 the MTP-only fallback
  • unsloth/GLM-5.3-Flash-FP8 weights (328 GB) — or LibertAIDAI/GLM-5.3-Flash-NVFP4 (194 GB) for ~23% more speed
  • incoai/GLM-5.3-Flash-DFlash2 drafter (2.34 GB) — worth ~65% decode; the 1.1 and 1.2 images run it out of the box
  • ~340 GB free per node for weights and the Triton cache
  • Root on every node (RDMA passthrough, memlock, page-cache flushing)

GLM-5.3-Flash served across four DGX Spark GB10 nodes (TP4 over dual 200G RoCE). Builds on tonyd2wild's NVFP4 1M-KV recipe for the attention route and FujitsuPolycom's TP4 Spark recipe for the NCCL ring profile.

02Build

The memory wall

GB10 is unified memory, and its NVRM allocator uses __GFP_RETRY_MAYFAIL: it requires actual MemFree and will not evict page cache. A node showing 45 GiB of cache and 2 GiB free fails a GPU allocation that a node with 45 GiB genuinely free serves. drop_caches moves 1–4 GiB free to 45–51 GiB immediately.

  • Flush page cache before every boot on every node; run a continuous low-rate flusher for the life of the deployment.
  • Pin the KV slab with --kv-cache-memory-bytes instead of letting the profiler size it — at FP8, weights are ~82 GB/rank and MemFree sits at 5–9 GB, the edge of the failure zone.
  • production-1.2 ships the pin at 12 GiB: −25% KV pool (2,252,384 → 1,688,835 tokens) for ~4 GB per rank of host headroom, which is what made CUDA graphs safe on the head node — MemAvailable 6.4–7.5 GB through a 128K prefill instead of 2.4–4.7.
  • Set a memory floor on every node and persist it in /etc/sysctl.d/: vm.min_free_kbytes=1048576, vm.dirty_bytes=536870912, vm.dirty_background_bytes=134217728, vm.dirty_expire_centisecs=500, vm.dirty_writeback_centisecs=100, vm.swappiness=10. Dirty pages are the one kind of cache drop_caches cannot free; capping them at 512 MiB and keeping 1 GiB genuinely free stops a download, log burst or checkpoint copy from starving the next GPU allocation. Do not go to 2 GiB: the higher watermark lowers MemAvailable by the same amount and puts a 121 GB node onto earlyoom's 2% line.
  • Even with the floor, do not run downloads or large writes on a serving node. The floor turns a certain crash into a throttled writer; it does not make the node a workstation.
  • Read the GPU KV cache size line rather than scaling the token count by the ratio — that arithmetic does not hold on this model (see the context section).

Checkpoint selection

unsloth/GLM-5.3-Flash-FP8 — 328 GB

  • vLLM-native: no tensor remapping, MTP head present, marlin is a valid FP8 MoE backend. The configuration carries over unchanged.

LibertAIDAI/GLM-5.3-Flash-NVFP4 — 194 GB

  • 23% faster across the board (110.7 vs 84.8 structured): decode is memory-bandwidth-bound and FP8 weights are twice the bytes.
  • Restores headroom — 27–37 GB MemFree per node instead of 5–9.

Per-node container

One container per node, host networking, RDMA passthrough. The fabric variables are the only site-specific values. NCCL_IB_HCA must list both RoCE devices: each Spark has two active 200G links, and pinning NCCL to one is pinning it to half the fabric.

docker run · per node
docker run -d --name glm53flash \
  --network host --ipc host --shm-size 32g --gpus all \
  --cap-add IPC_LOCK --ulimit memlock=-1:-1 \
  --memory 112g --memory-swap 112g \
  --device /dev/infiniband:/dev/infiniband \
  -v $HOME/.cache/huggingface:/cache/huggingface \
  -e HF_HOME=/cache/huggingface \
  # --- your fabric (per cluster / per node) ---
  -e NCCL_IB_HCA=<dev0>,<dev1>              # e.g. rocep1s0f0,roceP2p1s0f0 — list both
  -e NCCL_SOCKET_IFNAME=<your-fabric-iface> \
  -e GLOO_SOCKET_IFNAME=<your-fabric-iface> \
  -e VLLM_HOST_IP=<this-node-fabric-ip> \
  # --- address the fabric by subnet, not by GID index ---
  -e NCCL_IB_ROCE_VERSION_NUM=2 \
  -e NCCL_IB_ADDR_FAMILY=AF_INET \
  -e NCCL_IB_ADDR_RANGE=10.0.0.0/24 \
  # --- ring profile (FujitsuPolycom) ---
  -e NCCL_NET=IB -e NCCL_IB_DISABLE=0 \
  -e NCCL_ALGO=Ring -e NCCL_PROTO=LL,LL128,Simple \
  -e NCCL_SKIP_TREE_CONNECT=1 \
  -e NCCL_IB_SUBNET_AWARE_ROUTING=1 -e NCCL_IB_SUBNET_PREFIX_LEN=24 \
  -e NCCL_IB_MERGE_NICS=0 -e NCCL_P2P_LEVEL=SYS -e NCCL_NET_PLUGIN=none \
  -e NCCL_MAX_NCHANNELS=4 -e NCCL_MIN_NCHANNELS=4 \
  -e NCCL_CROSS_NIC=1 -e NCCL_CUMEM_ENABLE=0 -e NCCL_IGNORE_CPU_AFFINITY=1 \
  # --- housekeeping ---
  -e TORCH_USE_RTLD_GLOBAL=1 -e TORCHINDUCTOR_COMPILE_THREADS=1 \
  -e CUTE_DSL_ARCH=sm_121a -e TORCH_CUDA_ARCH_LIST=12.1a \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  aidendle94/sparkrun-vllm-glm53-gb10:production-1.2 \
  bash -lc 'unset VLLM_PREFIX_CACHE_RETENTION_INTERVAL; exec <the serve command below>'

The serve command

Start workers (ranks 1..N-1) headless first, then rank 0. Before any boot, run sync && echo 3 > /proc/sys/vm/drop_caches on every node — see the memory wall.

vllm serve · rank 0
vllm serve <path-to>/GLM-5.3-Flash-FP8 \
  --served-model-name <your-alias> --host 0.0.0.0 --port 8210 \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --nnodes 4 --node-rank $RANK --master-addr $HEAD_IP --master-port 29513 \
  --distributed-executor-backend mp --disable-custom-all-reduce \
  --distributed-timeout-seconds 240 --cpu-distributed-timeout-seconds 3600 \
  # functional parity — each of these fixes a silent client-facing gap
  --reasoning-parser glm45 \
  --tool-call-parser glm47 --enable-auto-tool-choice \
  --chat-template <path-to>/chat_template_mm.jinja \
  --default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "high"}' \
  # the SM90-NoPE + FA2 route: plain fp8 cache, real qk_rope_head_dim=0
  --kv-cache-dtype fp8_e4m3 \
  --block-size 2304 \
  --moe-backend marlin \
  --max-model-len 393216 --max-num-batched-tokens 8192 --max-num-seqs 16 \
  --enable-chunked-prefill \
  --gpu-memory-utilization 0.85 \
  --kv-cache-memory-bytes 12884901888 \
  --speculative-config '{"method":"dflash","model":"<path-to>/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7}'
  • --enforce-eager is gone in 1.2. CUDA graphs measured neutral on 1.0/1.1, but that reading was taken at the 16 GiB pin, where graphs pushed the head node into the earlyoom zone during long prefill; at 12 GiB they are worth +4–19% decode by regime and cost under 1 GB and ~30 s of capture. Prefill is unaffected either way.
  • --gpu-memory-utilization 0.85 is not arbitrary: 0.78 and 0.80 both starve the KV cache.

Published image contents

  • production-1.2 (2026-09-04): 1.1 plus CUDA graphs, the 12 GiB KV pin and a fused hyper-connection (mHC) Triton kernel on the prefill path. Same weights, same drafter, same flags otherwise; greedy answers agree with 1.1 on 30/30 keyed prompts.
  • production-1.1 is the eager predecessor: the SM90-NoPE-to-capability-12 attention route, the three functional fixes, and the DFlash2 overlay (drafter package, aux-capture patch, KV-page LCM patch, backend-registry patch).
  • production-1.0 is the same stack without DFlash2. It runs the checkpoint's native MTP head out of the box: use '{"method":"mtp","num_speculative_tokens":5}' and expect ~66 tok/s structured instead of 84.

03Deep dive

Attention backend selection

GLM5Next is NoPE (qk_rope_head_dim=0) — 45 layers: 34 KDA linear attention, 11 sparse MLA (kv_lora_rank=512, 64 heads, index_topk=2048, index_kpool=4). Two routes exist on SM121.

FLASHINFER_MLA_SPARSE_SM120 + fp8_ds_mla KV — not viable

  • Hardcodes a positional-embedding dimension of 64. On a NoPE model this requires faking pe_dim==64: rope padding via --hf-overrides, per-head q_b_proj padding for TP4 sharding, and custom kernels for the 2176-wide buffer.
  • The stack compiles and loads, then wedges the GPU.

SM90 NoPE MLA backend, capability 12, FA2 — used here

  • Accepts a plain bf16/fp8_e4m3 cache and a real qk_rope_head_dim=0.
  • Eliminates the rope padding, fp8_ds_mla, the q_b_proj padding and the 2176 kernels. This is what the published image ships.

Speculative decoding: DFlash2

Without speculation this stack runs 20.6 tok/s, flat across output regimes. DFlash2 (incoai/GLM-5.3-Flash-DFlash2) is the larger win: 110 vs 66 tok/s structured on the NVFP4 build, 5.06 accepted tokens per step vs the native MTP head's 2.81.

  • k=6 vs k=7: k=6 wins on freeform prose (40.4 vs 35.9 tok/s), loses on structured output. Ship 7 for agentic traffic, 6 for prose-heavy.
  • Native MTP head: optimum is k=5, not the k=4 most launchers use; ~66 tok/s structured. It costs 20% less KV pool (1.97M vs 1.58M tokens).
  • Reasoning effort: DFlash2 acceptance falls on reasoning-heavy output — predictable completions draft best.
  • Concurrency: DFlash2 is a single-stream latency win. At 8- and 12-way concurrency it matches MTP (122.6/152.2 vs 125.1/152.3) — saturated GPUs leave drafting no SMs.

Reasoning effort

The chat template opens with {%- set effective_reasoning_effort = reasoning_effort if reasoning_effort is defined and reasoning_effort in ['low', 'high'] else 'max' -%}. Three consequences:

  • Only `low` and `high` are honoured. Every other value falls to the else-branch. max works by accident; medium silently becomes Max.
  • Leaving it unset means Max, injected as a system line into every prompt.
  • That system line is emitted regardless of `enable_thinking`. The common '{"enable_thinking": false}' configuration renders a contradictory instruction on every request: reason at maximum effort, then a pre-closed empty thinking block forbidding it. The closed block wins, so behaviour is correct — but it is still contradictory.
effortreasoning produced
low0 words — thinking is on, and the model closes it immediately
high~36 words
max (or unset)~420 words

One puzzle prompt, three samples each. Max is more than 10× High, and scales with difficulty: at High, an easy factual question drew ~3 reasoning words while the puzzle drew ~36–47.

The shipped default is '{"enable_thinking": true, "reasoning_effort": "high"}' — stated explicitly rather than relying on the fall-through. Cost on short prompts is +0.5 s end-to-end (1.75 → 2.27 s). Raw decode is largely unaffected (84.1 → 89.3 tok/s structured), except highly predictable output like a repeated alphabet, which drops from 82.5 to 55.2 tok/s because reasoning prose is less draftable. Clients can still override per request with reasoning_effort: "low" or enable_thinking: false.

How long a context to ask for

The checkpoint's native limit is 1,048,576 tokens (text_config.max_position_embeddings), so anything up to 1M is inside the trained range — no rope scaling, no VLLM_ALLOW_LONG_MAX_MODEL_LEN fiction. A modest --max-model-len here is your own conservatism, not a model limit.

Going from 131,072 to 393,216 cost nothing measurable in decode: 83.6 / 83.0 / 76.3 / 28.8 tok/s after the change against 84.8 / 82.7 / 75.0 / 28.8 before. Per-token work does not grow with context, so neither does the cost of *allowing* more of it.

`--max-model-len`KV poolfull-length concurrency
131,0721,581,771 tokens12.07×
393,2162,252,384 tokens5.73×

With --kv-cache-memory-bytes pinned at exactly 16 GiB across both runs (the 1.1 pin; at the 1.2 pin of 12 GiB the 393,216 row is 1,688,835 tokens, 4.29×). The pool grew 42% from a change that should only have moved a ceiling: on this hybrid KDA-plus-MLA model the page geometry is negotiated against the mamba state, and that negotiation lands differently. Read your own GPU KV cache size line rather than predicting it.

Why this model does not get decode-context-parallel

DCP shards each layer's KV across ranks and roughly doubles the pool at DCP2; on the DeepSeek-V4 vision stack it does exactly that. On GLM-5.3-Flash it is not worth doing: only 11 of 45 layers hold real KV (the 34 KDA layers keep a small per-request recurrent state), which is why 2.25M tokens already fit with no sharding. What blocks it is the KpoolTailSpec — the one-block-per-request circular tail that feeds the KV-pool compressor — which is neither a full-attention, Mamba, nor replicated group, so the hybrid KV coordinator's DCP branch asserts. Making it replicated is the trick that made DSpark run under DCP on DeepSeek, but the GLM image sits on an older vLLM tree with none of the replicated-group plumbing, and its kpool compress and DFlash2 draft paths have no DCP awareness. Several days of porting for 2× on a pool that is not the constraint.

04Performance

Measured 2026-09-04. Four-node TP4 · FP8 · marlin MoE · DFlash2 k=7 · CUDA graphs · 12 GiB KV pin per rank · thinking on at High by default. Decode is a streamed 400-token completion per regime with completion_tokens accounting; prefill is unique-content cold prefill with a needle check. Image: aidendle94/sparkrun-vllm-glm53-gb10:production-1.2.

Decode88.5–89.7tok/ssingle stream, structured output, thinking off; 88.9–93.2 with thinking on at High (two runs)
Decode · code80.7tok/sthinking off; 75.3 on production-1.1
Prefill2,045–2,150tok/scold, 32K–128K depth, needle found — unchanged from 1.1
TTFT4 → 40s8K → 94K context; grows linearly
KV pool1.69Mtokensat 12 GiB/rank — 4.29 concurrent full-length contexts (2.25M at the 1.1 pin of 16 GiB)
Head-node headroom6.4–7.5GBMemAvailable on rank 0 through a 128K prefill; 2.4–4.7 GB on 1.1
Context393Ktokensverified with a 386,159-token prefill
Draft acceptance47.8%DFlash2 k=7, mixed traffic (95.4% code); unchanged in 1.2
Answer agreement30/30keyed prompts, 1.2 vs 1.1 greedy outputs
Decode by output regime — thinking offtok/s · single stream · higher is better
production-1.1 (eager, 16 GiB pin)production-1.2 (CUDA graphs, 12 GiB pin)
structured86.489.7+4%
alphabet86.289.6+4%
code75.380.7+7%
freeform prose28.533.8+19%

Same harness, same nodes, same day. CUDA graphs take launch overhead out of the decode step; the gain is largest where acceptance is lowest.

Decode by output regime — thinking on (High)tok/s · single stream · higher is better
production-1.1production-1.2
structured89.593.2+4%
alphabet76.788.1+15%
code79.378.8-1%
freeform prose31.233.1+6%

The shipped default. Reasoning prose is less draftable than the answer that follows it, so alphabet-style output no longer collapses the way it did on 1.1.

Cold prefill by depthtok/s · higher is better
production-1.1production-1.2
32K tokens2,1242,070-3%
128K tokens2,0932,045-2%

Within run-to-run noise (±3%). Prefill at TP4 is bound by attention, the tensor-parallel all-reduce and the MoE; graphs do not touch it.

Head-node MemAvailable at idle, by KV pinGB free on rank 0 · higher is safer
16 GiB pin, eager (1.1)2.4
16 GiB pin, CUDA graphs4.7
12 GiB pin, eager7.3
12 GiB pin, CUDA graphs (1.2)shipped7.5

GB10 is unified memory and earlyoom fires at 2% MemAvailable (~2.4 GB). The 16 GiB configurations touched that line during long prefill; at 12 GiB the floor through a 128K prefill is 6.4 GB.

KV pool by pintokens · higher is better
16 GiB per rank (1.1)2,252,384
12 GiB per rank (1.2)shipped1,688,835

−25% pool for ~4 GB per rank of host headroom. 4.29 full-length contexts still fit, and only 11 of 45 layers hold real KV on this model.

Changelog · production-1.1 → production-1.2

production-1.1production-1.2
CUDA graphsoff (--enforce-eager); graphs had measured neutralon — +4–19% decode by regime (structured 86.4 → 89.7, code 75.3 → 80.7, freeform 28.5 → 33.8 tok/s). The earlier "neutral" reading was taken at the 16 GiB pin, where graphs pushed the head node into the earlyoom zone during long prefill
--kv-cache-memory-bytes16 GiB per rank — KV pool 2,252,384 tokens (5.73×)12 GiB per rank — 1,688,835 tokens (4.29×); head-node MemAvailable 2.4–4.7 → 6.4–7.5 GB
Hyper-connection (mHC) kernelthree TileLang kernels per layerone fused Triton kernel for prefill tiles of 1,024+ tokens: 1.49× on that section in isolation, +1–2% end-to-end prefill; VLLM_MHC_FUSED_LARGE=0 disables it
Prefill2,073–2,124 tok/s at 32K–128K2,045–2,150 tok/s — unchanged within noise
Qualitykeyed 30/30keyed 30/30, greedy outputs agree with 1.1 on 30/30
Tried and rejected—deep_gemm MoE backend (−12% prefill, −19% decode); Triton FP8 W8A8 block-scaled MoE (4.7% relative error vs 0.6% for marlin); 16K prefill chunks (head-node OOM at the 16 GiB pin, untested at 12 GiB)
Weights, drafter, flagsunsloth FP8, DFlash2 k=7, thinking on at Highunchanged
Host memory floor (2026-09-10)kernel defaults: 45 MB minimum free, dirty pages up to 20% of RAM1 GiB minimum free, dirty pages capped at 512 MiB with 1 s writeback, swappiness 10 — after a download on the head killed the engine with cudaErrorNotPermitted on 2026-09-09
Concurrency validation (2026-09-10)single-stream benches only4 and 8 concurrent pp2048/tg128 (2,192 and 2,263 tok/s aggregate prefill) under a 12 GB page-cache write, 3×64K coherence soak, decode 88.5/88.4/79.3/33.2, quality 30/30, 32K/128K prefill 2,090/2,071 — CUDA graphs on

Same weights and nodes on both sides unless the row says otherwise.

Across recipes

Single-stream decode, by recipetok/s · higher is better
Inkling-2-Small · SGLang (experimental)60.0
DeepSeek-V4.1-Flash · SGLang (1.8)96.5
DeepSeek-V4.1-Flash (1.1, DCP2)60.8
GLM-5.3-Flash (1.2)shipped89.7
DeepSeek-V4-Flash (3.76.1)71.0
DeepSeek-V4-Flash-Vision (3.73)67.9

Different prompt sets, all single stream at temperature 0 with completion_tokens accounting: exact within a recipe, indicative across them. Inkling-2-Small · SGLang (experimental): code questions of 400 tokens (60–61), DSpark with the confidence gate, experts sharded by width; 56.6 at temperature 0.7 · DeepSeek-V4.1-Flash · SGLang (1.8): code prompts, DSpark block 5, RoCE collectives, early Engram staging, V4.1-Flash-1017; 113.7 counting, 34.9 prose · DeepSeek-V4.1-Flash (1.1, DCP2): code prompts, DSpark k=5, b12x MoE, DCP2 (77.5 on 1.0 with a different prompt set) · GLM-5.3-Flash (1.2): structured output, thinking off · DeepSeek-V4-Flash (3.76.1): mixed prompts, 8-rep mean · DeepSeek-V4-Flash-Vision (3.73): code prompts, DCP2 + DSpark

Cold prefill, by recipetok/s · unique content · higher is better
Inkling-2-Small · SGLang (experimental)2,096
DeepSeek-V4.1-Flash · SGLang (1.8)4,892
DeepSeek-V4.1-Flash (1.0)2,143
GLM-5.3-Flash (1.2)shipped2,070
DeepSeek-V4-Flash (3.76.1)2,200
DeepSeek-V4-Flash-Vision (3.73)2,380

Cold means the prompt has never been seen by the server; with prefix caching on, a repeated prompt is a cache hit and reads 3–25× higher. Inkling-2-Small · SGLang (experimental): 130K depth, cold (2,417–2,482 at ~15K) · DeepSeek-V4.1-Flash · SGLang (1.8): 32K depth, unique random words (3,848 at 128K) · DeepSeek-V4.1-Flash (1.0): 36K depth (1,940 at 132K) · GLM-5.3-Flash (1.2): 32K depth · DeepSeek-V4-Flash (3.76.1): 334K depth · DeepSeek-V4-Flash-Vision (3.73): 92K depth (2,088 at 334K)

Decode is flat with context: 91 → 96 tok/s from 805 to 93,534 tokens. 34 of 45 layers are KDA linear attention (O(1) in context); the 11 MLA layers are sparse at index_topk=2048, so per-token attention work stops growing past ~2K. Long contexts are cheap to serve and expensive to prefill.

05Notes

Three gaps that are silent until a client hits them

Missing flagSymptom
--reasoning-parser glm45Chain-of-thought leaks into message.content instead of the reasoning field
--tool-call-parser glm47 --enable-auto-tool-choiceRequests with tool_choice: auto return HTTP 400
--chat-template <mm template>Image requests return 500 — the checkpoint ships a text-only template

The server comes up healthy without these.

One node ran hot

One of the four units reset silently three times, each time one to three minutes into a sustained all-node long prefill, with nothing in the journal. Sampling every node every two seconds under identical load showed that unit's GPU 10 °C hotter and drawing ~25% more power than its twin, on identical firmware, with the board's critical trip at 104.8 °C. Capping that one GPU at 90% of its application clock (nvidia-smi -lgc 0,2176, persisted with a oneshot systemd unit) took it from 87–89 °C to 69 °C under the same load for a 1.2% cluster prefill cost. It has not reset since. Stream the sensor samples to another machine: a log written on the node that resets dies with it.

FP8 MLA with DFlash2 does not work

Operational caveats

  • Keep 1 GiB truly free and cap dirty pages on every node.GB10's driver allocates from genuinely free pages and never evicts cache. A writer can fill memory with dirty pages the flusher cannot drop, and the next GPU allocation fails with cudaErrorNotPermitted — which is how a checkpoint download on the head took the engine down seven seconds after it started. Six sysctls (memory-wall section) close it.
  • Benchmark concurrency in a window, never against production.The failure above surfaced under a four-request benchmark run against the live endpoint. Run concurrency tests with production down, from a flow that restores it and re-arms the watchdog.
  • Liveness is /health, never /v1/models./v1/models returns 200 with a dead engine; /health returns 503 on EngineDeadError.
  • Arm watchdogs only after the first healthy response.A TP4 load takes ~10 minutes; a watchdog counting failures from t=0 tears down all four ranks mid-boot and relaunches indefinitely. Container uptime is not a health signal.
  • Capture logs before teardown.vLLM v1 cannot recover a dead engine core — recovery means relaunching all four ranks, and docker rm -f destroys the evidence. NVRM failures surface in logs minutes after the event.
  • Count completion_tokens, not SSE chunks.Speculative decoding emits multiple tokens per chunk; chunk counting made a 3× speedup read as a 30% regression.
  • Benchmark with unique content.Prefix caching turns repeated prefill into a cache hit: 4,690 tok/s warmed vs 2,270 cold on the same 9K prefix. A 14K prompt is below the ~24K threshold where persistent_topk engages.
  • Verify the endpoint is idle before benchmarking.A busy alias costs roughly a third of decode. /metrics settles it: unchanged acceptance with lower throughput means the speculator is fine and the GPU is shared.
  • Arm the page-cache flusher idempotently.A per-boot flusher with a broken pre-kill accumulated 36 flushers per node across ~36 launches, each dropping caches every 20 s.
  • A full relaunch takes ~15 minutes on the FP8 checkpoint.Measured 963 s from docker run to a healthy /health on 2026-09-03. Size watchdog ready-timeouts accordingly; the fleet watchdog here uses 3600 s.
  • Do not hardcode NCCL_IB_GID_INDEX.RoCE GID tables renumber when NIC addresses change — a switch outage moved the entry from index 3 to 4 and every collective failed with ibv_modify_qp … ENODATA. Subnet addressing (NCCL_IB_ADDR_RANGE et al.) is immune.
© 2026 AidenLabBuilt on measured results, not vendor claims.