DeepSeek-V4-Flash-Vision on 4× DGX Spark

DeepSeek-V4-Flash-Vision-Exp served on four DGX Spark GB10 nodes: TP4 with decode-context-parallel 2, DSpark speculative decoding, thinking on, images in. The stack runs on the same sparkinfer B12X kernels as the text deployment; preprocessing, tower, routing and attention match upstream vLLM's vision support (PR #54566, merged 2026-09-02). Upstream has neither DCP nor speculative decoding for the vision variant. Image: aidendle94/sparkrun-vllm-ds4-gb10:production-3.73-vision.

Memory budget

GB10 memory is shared between host and GPU, and earlyoom on these nodes fires at 2% MemAvailable. The first "DSpark hang" investigated here was an earlyoom SIGTERM: at --gpu-memory-utilization 0.82 the boot left ~2.7 GiB free, one worker was killed mid-request, and the other three ranks waited on a collective that never came. 0.75 leaves 8–15 GiB per node through 334K prefill and 8-way image stress. Two rules follow.

• Clear the pinned KV size the launcher inherits from its version file (KV_CACHE_MEMORY_BYTES= set to empty). Left in place it caps the pool at ~26 GB per rank and hides the DCP capacity gain entirely.
• On a two-node TP2 layout each node carries half the weights plus the draft: 0.75 fails KV allocation outright, 0.82 leaves one rank below the minimum for 163K, 0.85 works. Budgets do not transfer between layouts; read the Available KV cache memory line per rank.
• Apply the host memory floor from the GLM recipe on every node (vm.min_free_kbytes=1048576, vm.dirty_bytes=536870912, vm.dirty_background_bytes=134217728, vm.dirty_expire_centisecs=500, vm.dirty_writeback_centisecs=100, vm.swappiness=10, persisted in /etc/sysctl.d/). The driver needs genuinely free pages and never evicts cache; dirty pages from any large write are what drop_caches cannot free, and a download on the head node killed a running engine that way on 2026-09-09.

Per-node deployment

One container per node, host networking, RDMA passthrough, the model and a writable JIT cache bind-mounted, driven by environment variables. Every node needs its own .env: the fabric address differs per node, and copying the head's file over a worker's makes that worker try to bind the head's address.

IMAGE=aidendle94/sparkrun-vllm-ds4-gb10:production-3.73-vision
MODEL_HOST_PATH=<path-to>/models--deepseek-ai--DeepSeek-V4-Flash-Vision-Exp
CACHE_HOST_PATH=<writable-jit-cache-dir>
SERVED_MODEL_NAME=deepseek-v4-vision
API_PORT=8300
NNODES=4  TP_SIZE=4
MASTER_ADDR=<head-fabric-ip>  MASTER_PORT=29511
VLLM_HOST_IP=<this-node-fabric-ip>          # differs per node
FABRIC_IFACE=<fabric-iface>  IB_HCA=<rdma-device>  IB_GID_INDEX=<resolved-at-boot>

# layout: TP4 across the nodes, DCP2 inside it
DCP_SIZE=2
DCP_COMM_BACKEND=ag_rs                       # +12% KV over a2a, decode within noise
KV_CACHE_MEMORY_BYTES=                       # empty on purpose: unpins the ~26 GB cap
GPU_MEMORY_UTILIZATION=0.75                  # 0.82 gets a worker killed by earlyoom
MAX_MODEL_LEN=393216  MAX_NUM_SEQS=8  MAX_NUM_BATCHED_TOKENS=8192

# speculative decoding
VLLM_ENABLE_DSPARK=1  VLLM_ENABLE_MTP=0
NUM_SPECULATIVE_TOKENS=6
DSPARK_DRAFT_SAMPLE_METHOD=greedy  DSPARK_SPS_CURVE=auto
MAX_CUDAGRAPH_CAPTURE_SIZE=56                # = MAX_NUM_SEQS × (spec tokens + 1)
VLLM_CUDAGRAPH_MODE=FULL_AND_PIECEWISE

# prefill gather (required for images under DCP)
VLLM_DSV4_CKV_GATHER=1
VLLM_DSV4_CKV_GATHER_MAX_TOKENS=393216
VLLM_DSV4_CKV_ALLOW_IMAGES=1

ENABLE_PREFIX_CACHING=1
VLLM_DSV4_DCP_PREFIX_CACHE=1                 # unset = 3.72 behaviour
VISION_MHC=1

vllm serve <model-snapshot> \
  --served-model-name deepseek-v4-vision --host 0.0.0.0 --port 8300 \
  --trust-remote-code --distributed-executor-backend mp \
  --nnodes 4 --node-rank $RANK --master-addr $HEAD_IP --master-port 29511 \
  --tensor-parallel-size 4 \
  --decode-context-parallel-size 2 --dcp-comm-backend ag_rs \
  --attention-backend B12X_MLA_SPARSE --moe-backend b12x --linear-backend b12x \
  --kv-cache-dtype fp8 --block-size 256 \
  --gpu-memory-utilization 0.75 \
  --max-model-len 393216 --max-num-seqs 8 --max-num-batched-tokens 8192 \
  --max-cudagraph-capture-size 56 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"],"cudagraph_capture_sizes":[1,2,3,4,5,6,7,11,15,...,56]}' \
  --async-scheduling --no-scheduler-reserve-full-isl --enable-chunked-prefill --enable-prefix-caching \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' \
  --default-chat-template-kwargs.thinking=true \
  --speculative-config '{"method":"dspark","num_speculative_tokens":6,"draft_sample_method":"greedy", ...}' \
  --disable-chunked-mm-input --limit-mm-per-prompt '{"image": 4}'

1. Drop page caches on every node, start the three workers, then the head. Healthy in ~10 minutes: weight load, draft load, KV allocation, 74 s of graph capture, then the DSpark SPS-curve profile.
2. Read two lines from rank 0 before trusting anything: GPU KV cache size (expect ~3.5M tokens) and the acceptance length from a decode bench (expect 3.5–3.8).
3. Send one image first. The first image request compiles the partition kernel (~6 s); later ones take about a second for a 350-token image prompt.
4. Send one long prompt twice. vllm:prefix_cache_hits_total in /metrics should climb by the prompt length rounded down to 512 tokens, and the second answer should start within about a second. Zero hits means the container never received VLLM_DSV4_DCP_PREFIX_CACHE=1.

{"model": "deepseek-v4-vision",
 "messages": [{"role": "user", "content": [
   {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
   {"type": "text", "text": "What animal is this? Two sentences."}]}],
 "max_tokens": 160, "temperature": 0,
 "chat_template_kwargs": {"thinking": false}}

# → "This is a brown and black tabby cat with striking green eyes. It is sitting
#    upright on a textured white ledge, with blurred bare branches visible in the
#    background."   (352 prompt tokens, 1.2 s warm)

Figures are from the live four-node cluster with unique-content prompts and completion_tokens accounting; acceptance lengths and cache hits come from the /metrics counters. Prefix-cache figures are from production-3.73-vision on 2026-09-04; profile comparison rows are from 3.72-vision on the same nodes.

Vision port details

• Preprocessing and layout. Resize solver, patch grid, the N-layout row-pair permutation and the five sentinel tokens (start, pad, image, newline, end) are bit-identical to the checkpoint's reference code, verified over odd row counts, extreme aspect ratios and sub-minimum images. Sentinels live past the text vocabulary and are sanitized before every embedding lookup.
• Routing for image tokens. Image rows route on scores + bias_vl with the unbiased scores as weights; text rows keep the stock kernel path. Hash-routed layers need the score path for image rows, and their tid2eid table is exactly vocab_size rows tall, so an unguarded lookup on a sentinel id is out-of-bounds memory or a device assert.
• Sliding-window visibility. Image tokens see the whole image (not the causal window) plus the trailing text window. The B12X dual-cache prefill is pinned to a 128-wide window, so extra slots run as separate 128-wide attention partitions merged exactly in fp32 from base-2 log-sum-exps; measured max error against a single-partition reference is 2.4e-7.
• Multi-image batches. The tower runs per-item grids; concatenating patches across requests tripped a rotary size mismatch under concurrency.

DSpark and DCP together

The stock stack refuses this combination twice: the launcher exits with "DSpark requires DCP_SIZE=1" and the sliding-window metadata builder asserts "Non-causal DSpark SWA does not support DCP". The verify pass is causal and already has full DCP kernels. The non-causal path belongs to the draft: three full decoder layers, sliding-window only, whose block forward attends over the whole speculative block at once. Its KV cache was landing in the target's sharded sliding-window group, where the draft's slot math is wrong.

• The DSpark SPS-curve profiler was the first DCP consumer of the generic dummy-run branch, which built batches without rank-local sequence lengths; the CUDA-graph capture path computes them itself, which is why capture passed and the profiler asserted. The dummy branch now derives them the way real batches do.
• --dcp-comm-backend ag_rs over a2a: same decode within noise, +12% KV (3.51M vs 3.13M tokens) because all-to-all reserves more scratch.
• Prefix caching under DCP was off in 3.72-vision by design; 3.73-vision turns it on. What was actually wrong is below.

Prefix caching under DCP (3.73-vision)

In 3.72-vision the KV coordinator carried a blanket guard for this model's MLA/SWA hybrid layout, so --enable-prefix-caching was accepted and inert and every request paid a cold prefill. The guard hid a bug: under DCP each rank holds one local block per block_size × dcp global tokens, and the sliding-window manager registered hashes at that scaled size but looked them up at the unscaled 64-token block, so nothing it inserted could be found. The coordinator also handed every sliding-window group a DCP size of 1. Two changes fix it: lookup and the sparse-retention mask use the manager's DCP-scaled block size (replicated groups scale by 1), and the coordinator passes the real DCP size to every attention group and asserts a hit ends on a scheduler-block boundary, 512 tokens at DCP2.

• Hits come in 512-token blocks. The tail past the last boundary is recomputed every time, and every sliding-window and compressor-state group has to hold its window tail at that boundary. A 1.3K prompt gets 77% of itself from cache, a 92K prompt 100%.
• DSpark keeps speculating on cached prefixes. The draft only requires the cached length to be a multiple of every kernel block size, which a 512-aligned hit always is. Acceptance on warm requests measured 3.3–3.9 tokens per step, the same as cold.
• Opt-in. VLLM_DSV4_DCP_PREFIX_CACHE=1 turns it on; unset, 3.73-vision behaves like 3.72. The scheduler logic was exercised offline on CPU with a synthetic copy of this model's KV groups at DCP 1, 2 and 4.

The prefill gather

Stock DCP prefill on this model all-gathers query heads and reduces log-sum-exps across ranks per layer, about a gigabyte per 8K-token chunk at DCP4; cold 334K prefill collapsed to 1,046 tok/s at DCP4 and 1,440 at DCP2. This port gathers each rank's slice of the compressed and sliding-window caches into a rank-local arena in global token order once per layer, then runs the unchanged DCP=1 attention against a synthetic contiguous block table. Cold 334K prefill: 2,184 tok/s at DCP4, 2,119 at DCP2, byte-identical outputs with the gather on or off.

Images under DCP need the gather: image tokens must read slots that stock sharded prefill drops, so the stack refuses image requests under DCP unless the gather is enabled and images are explicitly allowed (VLLM_DSV4_CKV_GATHER=1, VLLM_DSV4_CKV_ALLOW_IMAGES=1).

Three profiles measured on the same nodes and image; the combined one is what ships.

One node ran hot

One of the four units reset silently three times, always one to three minutes into an all-node 334K prefill — no shutdown record, no kernel error, no thermal message; the journal just stops. Sampling every node every two seconds under identical load explained it: that unit's GPU ran at 87–89 °C and 63–70 W where its twin ran 77–80 °C and 51–59 W, on identical firmware. The critical trip is 104.8 °C, and a firmware trip looks exactly like a power cut. Capping that GPU at 90% of its application clock (2,176 MHz) brought it to 69 °C and 43 W under the same load for a 1.2% cluster prefill cost.