AidenLabrecipe 01
MEASURED · 2026-11-15
MEASURED·8 min read·Aug 20, 2026

DeepSeek-V4-Flash on 4× DGX Spark

DeepSeek-V4-Flash-0731 on four DGX Spark GB10 nodes (TP4 over 200G RoCE): ~71 tok/s decode, 393K context, thinking on by default.

4× DGX Spark GB10 · TP4/EP4 · vLLM 3.76.1 · aidendle94/sparkrun-vllm-ds4-gb10:production-3.76.1 · measured Aug 20, 2026

  • DeepSeek-V4-Flash
  • vLLM
  • multi-node
  • TP4
  • DGX Spark
  • FP8
  • MoE
  • long context
~2,200tok/sPrefillcold at 334K depth
~71tok/sDecodesingle stream, mixed prompts, 8-rep mean
393KtokensContextverified with a 380K-token prefill

01Overview

What you need
  • 4× DGX Spark GB10 nodes, fabric-linked (200G RoCE or InfiniBand)
  • aidendle94/sparkrun-vllm-ds4-gb10:production-3.76.1 image (Docker Hub)
  • Hugging Face token with access to DeepSeek-V4-Flash-0731 weights
  • ~120 GB free per node for weights and the JIT kernel cache
  • Root on every node (RDMA passthrough and memlock)

DeepSeek-V4-Flash-0731 served on four DGX Spark GB10 nodes over 200G RoCE, TP4. Image: aidendle94/sparkrun-vllm-ds4-gb10:production-3.76.1.

02Build

Image composition

The image bakes in ~50 tuned defaults (AOT compile, V2 model runner, flashinfer sampler, PCIe allreduce, pinned NCCL, b12x MoE backend). 3.76.1 adds four fixes over 3.75, each shipped as a FROM-image-plus-COPY overlay:

  • CUDA-graph padding no longer contaminates request slot 0 of the draft sampler (padding rows get a −1 sentinel and are masked in the gumbel kernels).
  • Draft gumbel noise moved to a Philox range disjoint from the rejection sampler (offset 1<<30).
  • Blocking CUDA events in the draft-tokens handler instead of spin-polling the driver lock.
  • The DSV4 chat-template kwargs (thinking, reasoning_effort) named explicitly in the tokenizer signature so serving-side filters cannot drop them.

Per-node container

One container per node, host networking, RDMA passthrough. Resolve NCCL_IB_GID_INDEX at boot: RoCE GID tables renumber when NIC addresses change (index 3-4-3 within a day here). Look up the index of your fabric IP RoCE-v2 entry under /sys/class/infiniband/<dev>/ports/1/gids/.

docker run · per node
docker run -d --name ds4f \
  --network host --ipc host --shm-size 10gb --gpus all \
  --cap-add IPC_LOCK --ulimit memlock=-1:-1 \
  --device /dev/infiniband:/dev/infiniband \
  -v $HOME/.cache/huggingface:/cache/huggingface \
  -e HF_HOME=/cache/huggingface \
  # --- your fabric (per cluster / per node) ---
  -e NCCL_IB_HCA=<your-rdma-device>            # e.g. rocep1s0f0
  -e NCCL_SOCKET_IFNAME=<your-fabric-iface>    # e.g. enp1s0f0np0
  -e GLOO_SOCKET_IFNAME=<your-fabric-iface> \
  -e NCCL_IB_GID_INDEX=<resolved-at-boot> \
  -e NCCL_MAX_NCHANNELS=4 -e NCCL_MIN_NCHANNELS=4 \
  -e VLLM_HOST_IP=<this-node-fabric-ip> \
  # --- perf pins (override image defaults; validated +16% together) ---
  -e VLLM_DSPARK_FP8_LM_HEAD=0 \
  -e VLLM_DSPARK_MARKOV_W2_FP8=0 \
  -e VLLM_DSPARK_REPLICATE_MARKOV_W1=0 \
  -e VLLM_DSV4_INDEXER_SP=1                    # prefill: indexer sequence-parallel
  # --- housekeeping ---
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  aidendle94/sparkrun-vllm-ds4-gb10:production-3.76.1 \
  <the vllm serve command below>

The serve command

  1. Start workers (ranks 1..N-1) first, wait ~10 seconds, then start rank 0.
  2. After another stack has run, sync && echo 3 > /proc/sys/vm/drop_caches on every node first — otherwise a rank can hang at weight load with no error.
  3. The first boot pays AOT compile and flashinfer autotune costs (both cached). Later boots take ~7 minutes.
vllm serve · rank 0
vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --served-model-name <your-alias> --host 0.0.0.0 --port 8210 \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --nnodes 4 --node-rank $RANK --master-addr $HEAD_IP --master-port 29512 \
  --distributed-executor-backend mp \
  --distributed-timeout-seconds 3600 --cpu-distributed-timeout-seconds 3600 \
  --attention-backend FLASHINFER_MLA_SPARSE_DSV4 --enable-flashinfer-autotune \
  --kv-cache-dtype fp8 --block-size 256 --load-format auto \
  --max-model-len 393216 --max-num-batched-tokens 4096 --max-num-seqs 128 \
  --max-cudagraph-capture-size 128 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --no-scheduler-reserve-full-isl --enable-chunked-prefill \
  --long-prefill-token-threshold 3072 --enable-prefix-caching \
  --gpu-memory-utilization 0.8 \
  --tool-call-parser deepseek_v4 --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":" thinking","reasoning_end_str":" response"}' \
  --default-chat-template-kwargs.thinking=true \
  --default-chat-template-kwargs.reasoning_effort=max \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}'

03Performance

Measured 2026-08-20. Four-node TP4, no DCP, DSpark speculative decoding, prefix caching on. Figures are from the measured-performance section of the 3.76.1 recipe; decode is an 8-repetition mean. Image: aidendle94/sparkrun-vllm-ds4-gb10:production-3.76.1.

Decode~71tok/ssingle stream, mixed prompts, 8-rep mean
Aggregate~250tok/s16-way concurrency
Prefill~2,200tok/scold at 334K depth
Warm prefill5–8Ktok/srepeated context via prefix caching
Context393Ktokensverified with a 380K-token prefill
Draft acceptance0.55DSpark, 5 speculative tokens

Across recipes

Single-stream decode, by recipetok/s · higher is better
Inkling-2-Small · SGLang (experimental)60.0
DeepSeek-V4.1-Flash · SGLang (1.8)96.5
DeepSeek-V4.1-Flash (1.1, DCP2)60.8
GLM-5.3-Flash (1.2)89.7
DeepSeek-V4-Flash (3.76.1)shipped71.0
DeepSeek-V4-Flash-Vision (3.73)67.9

Different prompt sets, all single stream at temperature 0 with completion_tokens accounting: exact within a recipe, indicative across them. Inkling-2-Small · SGLang (experimental): code questions of 400 tokens (60–61), DSpark with the confidence gate, experts sharded by width; 56.6 at temperature 0.7 · DeepSeek-V4.1-Flash · SGLang (1.8): code prompts, DSpark block 5, RoCE collectives, early Engram staging, V4.1-Flash-1017; 113.7 counting, 34.9 prose · DeepSeek-V4.1-Flash (1.1, DCP2): code prompts, DSpark k=5, b12x MoE, DCP2 (77.5 on 1.0 with a different prompt set) · GLM-5.3-Flash (1.2): structured output, thinking off · DeepSeek-V4-Flash (3.76.1): mixed prompts, 8-rep mean · DeepSeek-V4-Flash-Vision (3.73): code prompts, DCP2 + DSpark

Cold prefill, by recipetok/s · unique content · higher is better
Inkling-2-Small · SGLang (experimental)2,096
DeepSeek-V4.1-Flash · SGLang (1.8)4,892
DeepSeek-V4.1-Flash (1.0)2,143
GLM-5.3-Flash (1.2)2,070
DeepSeek-V4-Flash (3.76.1)shipped2,200
DeepSeek-V4-Flash-Vision (3.73)2,380

Cold means the prompt has never been seen by the server; with prefix caching on, a repeated prompt is a cache hit and reads 3–25× higher. Inkling-2-Small · SGLang (experimental): 130K depth, cold (2,417–2,482 at ~15K) · DeepSeek-V4.1-Flash · SGLang (1.8): 32K depth, unique random words (3,848 at 128K) · DeepSeek-V4.1-Flash (1.0): 36K depth (1,940 at 132K) · GLM-5.3-Flash (1.2): 32K depth · DeepSeek-V4-Flash (3.76.1): 334K depth · DeepSeek-V4-Flash-Vision (3.73): 92K depth (2,088 at 334K)

04Notes

Two configuration mistakes to avoid

The fp8 draft-head default costs 16%

  • Draft-head proposals were historically quantized to fp8; the code comment recorded ~93% top-6 overlap vs bf16.
  • Overriding two env vars gave +16% throughput and +0.09 acceptance after one restart. Both are in the recipe below.

Benchmark from the production config

  • A launcher adapted from a smoke-test script carried the wrong attention backend and no prefill machinery: deep prefill read 681 tok/s.
  • With the production compose config, cold 334K prefill went from 558 s to 154 s.

Operational caveats

Operational caveats

  • Thinking text lands in message.reasoning, not reasoning_content.Clients written against newer vLLM must read both fields.
  • Reasoning-config needs explicit thinking / </think> strings.Empty strings break reasoning-token-ID initialization on this fork.
  • RoCE GID indices renumber when NIC addresses change.Resolve the v2 fabric GID at boot on each node.
  • Boot with cold page caches after another stack has run.Otherwise a rank hangs at weight load with no error.
  • Prefix caching invalidates repeated-text benchmarks.Later prefill runs cache-hit earlier ones; measure cold with unique text.
© 2026 AidenLabBuilt on measured results, not vendor claims.