Hugging Face token with access to DeepSeek-V4-Flash-0731 weights
~120 GB free per node for weights and the JIT kernel cache
Root on every node (RDMA passthrough and memlock)
DeepSeek-V4-Flash-0731 served on four DGX Spark GB10 nodes over 200G RoCE, TP4. Image: aidendle94/sparkrun-vllm-ds4-gb10:production-3.76.1.
02Build
Image composition
The image bakes in ~50 tuned defaults (AOT compile, V2 model runner, flashinfer sampler, PCIe allreduce, pinned NCCL, b12x MoE backend). 3.76.1 adds four fixes over 3.75, each shipped as a FROM-image-plus-COPY overlay:
CUDA-graph padding no longer contaminates request slot 0 of the draft sampler (padding rows get a −1 sentinel and are masked in the gumbel kernels).
Draft gumbel noise moved to a Philox range disjoint from the rejection sampler (offset 1<<30).
Blocking CUDA events in the draft-tokens handler instead of spin-polling the driver lock.
The DSV4 chat-template kwargs (thinking, reasoning_effort) named explicitly in the tokenizer signature so serving-side filters cannot drop them.
Per-node container
One container per node, host networking, RDMA passthrough. Resolve NCCL_IB_GID_INDEX at boot: RoCE GID tables renumber when NIC addresses change (index 3-4-3 within a day here). Look up the index of your fabric IP RoCE-v2 entry under /sys/class/infiniband/<dev>/ports/1/gids/.
Measured 2026-08-20. Four-node TP4, no DCP, DSpark speculative decoding, prefix caching on. Figures are from the measured-performance section of the 3.76.1 recipe; decode is an 8-repetition mean. Image: aidendle94/sparkrun-vllm-ds4-gb10:production-3.76.1.
Decode~71tok/ssingle stream, mixed prompts, 8-rep mean
Aggregate~250tok/s16-way concurrency
Prefill~2,200tok/scold at 334K depth
Warm prefill5–8Ktok/srepeated context via prefix caching
Context393Ktokensverified with a 380K-token prefill
Draft acceptance0.55DSpark, 5 speculative tokens
Across recipes
Single-stream decode, by recipetok/s · higher is better
Inkling-2-Small · SGLang (experimental)60.0
DeepSeek-V4.1-Flash · SGLang (1.8)96.5
DeepSeek-V4.1-Flash (1.1, DCP2)60.8
GLM-5.3-Flash (1.2)89.7
DeepSeek-V4-Flash (3.76.1)shipped71.0
DeepSeek-V4-Flash-Vision (3.73)67.9
Different prompt sets, all single stream at temperature 0 with completion_tokens accounting: exact within a recipe, indicative across them. Inkling-2-Small · SGLang (experimental): code questions of 400 tokens (60–61), DSpark with the confidence gate, experts sharded by width; 56.6 at temperature 0.7 · DeepSeek-V4.1-Flash · SGLang (1.8): code prompts, DSpark block 5, RoCE collectives, early Engram staging, V4.1-Flash-1017; 113.7 counting, 34.9 prose · DeepSeek-V4.1-Flash (1.1, DCP2): code prompts, DSpark k=5, b12x MoE, DCP2 (77.5 on 1.0 with a different prompt set) · GLM-5.3-Flash (1.2): structured output, thinking off · DeepSeek-V4-Flash (3.76.1): mixed prompts, 8-rep mean · DeepSeek-V4-Flash-Vision (3.73): code prompts, DCP2 + DSpark
Cold prefill, by recipetok/s · unique content · higher is better
Inkling-2-Small · SGLang (experimental)2,096
DeepSeek-V4.1-Flash · SGLang (1.8)4,892
DeepSeek-V4.1-Flash (1.0)2,143
GLM-5.3-Flash (1.2)2,070
DeepSeek-V4-Flash (3.76.1)shipped2,200
DeepSeek-V4-Flash-Vision (3.73)2,380
Cold means the prompt has never been seen by the server; with prefix caching on, a repeated prompt is a cache hit and reads 3–25× higher. Inkling-2-Small · SGLang (experimental): 130K depth, cold (2,417–2,482 at ~15K) · DeepSeek-V4.1-Flash · SGLang (1.8): 32K depth, unique random words (3,848 at 128K) · DeepSeek-V4.1-Flash (1.0): 36K depth (1,940 at 132K) · GLM-5.3-Flash (1.2): 32K depth · DeepSeek-V4-Flash (3.76.1): 334K depth · DeepSeek-V4-Flash-Vision (3.73): 92K depth (2,088 at 334K)
04Notes
Two configuration mistakes to avoid
The fp8 draft-head default costs 16%
Draft-head proposals were historically quantized to fp8; the code comment recorded ~93% top-6 overlap vs bf16.
Overriding two env vars gave +16% throughput and +0.09 acceptance after one restart. Both are in the recipe below.
Benchmark from the production config
A launcher adapted from a smoke-test script carried the wrong attention backend and no prefill machinery: deep prefill read 681 tok/s.
With the production compose config, cold 334K prefill went from 558 s to 154 s.
Operational caveats
Operational caveats
Thinking text lands in message.reasoning, not reasoning_content.Clients written against newer vLLM must read both fields.
Reasoning-config needs explicit thinking / </think> strings.Empty strings break reasoning-token-ID initialization on this fork.
RoCE GID indices renumber when NIC addresses change.Resolve the v2 fabric GID at boot on each node.
Boot with cold page caches after another stack has run.Otherwise a rank hangs at weight load with no error.