DeepSeek-V4.1-Flash on 4× DGX Spark with SGLang

DeepSeek-V4.1-Flash served on four DGX Spark GB10 nodes with SGLang: TP4 with expert parallelism 4, the dsv4 attention backend, FlashInfer's MXFP4 MoE runner, DSpark speculative decoding with the checkpoint's own draft layers at block size 5, tools, images, thinking on at reasoning_effort max by default. The engine is lmsysorg/sglang:dev-dsv41, the SGLang team's own DeepSeek-V4.1 image, unmodified. Everything a Spark needs on top is an overlay of import hooks in one MIT repository — about 1,500 lines with the launcher — so the image can be rebuilt on a newer upstream tag without touching them, and the stock code paths stay available in the same image. On 2026-09-17 it replaced the vLLM stack of the previous recipe as production on this fleet: same port, same model names, same clients.

Per-node deployment

One container per node, host networking, RDMA passthrough, the model directory on NFS (the node that holds the checkpoint mounts it locally and reads its Engram rows straight from the shards; the others read their node-local copies). Workers start first, the head last; the launcher probes each node's RoCE-v2 GID index, refuses to start a rank while any GPU is still held by a previous engine, and prints the four commands with --dry-run. Rank 0 below; the workers differ in --node-rank, HOST_IP and their Engram source.

docker run -d --name sgldsv41 --restart no --network host --ipc host --cap-add IPC_LOCK --gpus all \
  --shm-size 32g --memory 112g --memory-swap 112g --ulimit memlock=-1:-1 --ulimit stack=67108864 \
  --device /dev/infiniband:/dev/infiniband --oom-score-adj 500 \
  -v <hf-repo-dir>:/models/repo:ro -v <writable-cache-dir>:/root/.cache \
  -v <engram-local-dir>:/engram-local:ro -e SPARK_ENGRAM_DIR=/engram-local \
  # the node that holds the checkpoint reads its rows from the shards instead of a copy:
  #   -e SPARK_ENGRAM_DIR=/models/repo/snapshots/<snapshot> -e SPARK_ENGRAM_ROWS=1:<lo>:<hi>,14:<lo>:<hi>
  -e SPARK_ENGRAM_MODE=hostnode -e SPARK_ENGRAM_EARLY=1 -e SPARK_ENGRAM_PREFETCH=1 -e SPARK_PAGE_CACHE_RELEASE=1 \
  -e SPARK_LATE_TAIL_SKIP=1 -e SPARK_LATE_TAIL_SKIP_ALL=1 -e SPARK_SM120_REAL_HEADS=1 -e SPARK_ENGRAM_THREADS=64 -e SPARK_ENGRAM_MAX_IDS=65536 -e SPARK_MXFP8_BACKEND=b12x \
  -e SPARK_SERVED_ALIASES=<other-model-ids,comma-separated> \
  -e SPARK_ROCE_AR=1 -e SPARK_ROCE_AR_MAX=1MB -e SPARK_ROCE_AG_MAX=16MB \
  -e B12X_ROCE_HCA=<rdma-devices> -e B12X_ROCE_GID_INDEX=<this-node-roce-v2-gid-index> -e B12X_ROCE_SPIN_LIMIT=300000000 \
  -e SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=0 -e SGLANG_FLASHINFER_MOE_FUSED_FINALIZE=0 -e SGLANG_DSV41_REASONING_EFFORT=100 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False -e CUDA_DEVICE_ORDER=PCI_BUS_ID -e HOST_IP=<this-node-fabric-ip> \
  -e NCCL_NET=IB -e NCCL_IB_DISABLE=0 -e NCCL_IB_HCA=<rdma-devices> -e NCCL_SOCKET_IFNAME=<fabric-iface> -e GLOO_SOCKET_IFNAME=<fabric-iface> \
  -e NCCL_IB_ROCE_VERSION_NUM=2 -e NCCL_IB_ADDR_FAMILY=AF_INET -e NCCL_IB_ADDR_RANGE=<fabric-subnet>/24 -e NCCL_IB_SUBNET_PREFIX_LEN=24 -e NCCL_IB_GID_INDEX=<gid-index> \
  -e NCCL_CROSS_NIC=1 -e NCCL_IB_MERGE_NICS=0 -e NCCL_IB_SUBNET_AWARE_ROUTING=1 -e NCCL_CUMEM_ENABLE=0 -e NCCL_P2P_DISABLE=1 -e NCCL_SHM_DISABLE=1 \
  -e NCCL_BUFFSIZE=1048576 -e NCCL_LL128_BUFFSIZE=262144 -e NCCL_PROTO=^LL128 -e NCCL_MAX_NCHANNELS=8 \
  --entrypoint python3 aidendle94/sparkrun-sglang-dsv41-gb10:production-1.7.1 -m sglang.launch_server \
    --model-path /models/repo/snapshots/<snapshot> --served-model-name deepseek-v4.1-flash --trust-remote-code --load-format safetensors \
    --tp 4 --ep-size 4 --nnodes 4 --node-rank 0 --dist-init-addr <head-fabric-ip>:20100 --host 0.0.0.0 --port 8210 \
    --attention-backend dsv4 --moe-runner-backend flashinfer_mxfp4 --fp8-gemm-backend flashinfer_cutlass \
    --mem-fraction-static 0.80 --chunked-prefill-size 2048 --context-length 524288 --max-running-requests 16 --cuda-graph-max-bs-decode 16 \
    --random-seed 0 --enable-decoder-swa-bounded-replay --speculative-algorithm DSPARK --speculative-dspark-block-size 5 \
    --tool-call-parser deepseekv41 --reasoning-parser deepseek-v41 --watchdog-timeout 1800 --min-free-slots-delay 1 \
    --default-chat-template-kwargs '{"thinking":true}'

1. Pull the image on every node (arm64; or docker build the repository's Dockerfile on lmsysorg/sglang:dev-dsv41 — it compiles the row store, vendors the b12x RoCE runtime and runs the CPU tests). Export the checkpoint's HuggingFace repo directory from the node that holds it and mount it on the others.
2. Make each rank's node-local Engram rows with the vLLM recipe's engram_local.py — one contiguous range per rank so the four ranges partition each table; the hook reads the range from the copy's manifest. The node with the checkpoint skips the copy and passes its range as SPARK_ENGRAM_ROWS.
3. --fp8-gemm-backend flashinfer_cutlass is required for the MXFP8 route (it selects the block-scaled path the hook wraps); the b12x kernel refuses the K = 576 projections and those fall back to CUTLASS per shape.
4. Healthy in ~10 minutes. Check four lines before trusting it: DSV4 memory calculation … full_token= (the KV pool, 4.2–6.6M depending on the boot), RoCE collectives on: world=4, Engram local layer=1 … gather_us_per_call near 1,000 at idle, and free -g on every node.
5. Run something that checks answers: count to 20 (exact), a needle at 32K and 128K, an image, a tool call. Then arm the watchdog (launch/watchdog.sh: /health plus a request probe with a busy grace, orchestrated relaunch with pre-teardown log capture).

curl -s http://<head>:8210/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "deepseek-v4.1-flash", "max_tokens": 300, "temperature": 0,
  "messages": [{"role": "user", "content": "Is 91 a prime number? Answer briefly."}],
  "chat_template_kwargs": {"thinking": true, "reasoning_effort": 25}}'
# "thinking": false turns it off; reasoning_effort: low 25 / high 50 / xhigh 75 / max 100 or any integer 1–100

What stock SGLang does on a Spark, and what the overlay changes

Stock dev-dsv41 loads the two Engram tables into pinned host memory — 203 GB, the same wall the vLLM recipe hit, since on a Spark host memory is the GPU pool — and two of its SM120 paths were written for a page size and an indexer schedule that V4.1's ratio-1/2 attention sources do not use, so the first boots died in graph capture and in the sparse prefill. Five 40-minute windows fixed one thing each; two more brought the collectives and the Engram gather cost over from the vLLM work. This is what ended up in the overlay, each piece behind an import hook that runs once, right after its target module loads.

Two ways to move the rows

The overlay ships two Engram implementations behind one switch (SPARK_ENGRAM_MODE). The default gathers each layer's rows inside the CUDA graph: a host-function node calls the C row store while the GPU waits, so nothing has to know the tokens ahead of time — which matters because SGLang's DSpark verify step gets its draft tokens on the GPU. The alternative is a port of our vLLM design: a hook before every forward hashes the batch with the model's own kernel, syncs once, gathers both layers' rows with a Python thread pool and stages them, and the embedding reads them by position. It needs no C library. Measured on the same nodes the same day, one boot each:

production-1.7 → 1.7.1 — prompt lookup, and Claude Code's minute-long turns

1.7 (2026-09-28) added prompt-lookup drafting on top of DSpark: when the answer repeats its own context — a file being edited, JSON being restated, a passage being quoted — the continuation of the earlier occurrence is verified along with DSpark's block. DSpark keeps drafting its 5 tokens; a row whose block continues an earlier 8-token match gets up to 10 more tokens of it, and only that row verifies the longer block (SGLang's compact ragged verify, graphs captured at 6b / 6b+10 / 6b+20 tokens). Putting lookup tokens in place of DSpark's lost; DSpark already takes its whole block on most copied text, so the gain is past its block.

It ran as production for one day and is now off by default (WIDE=1 turns it on). Twice the fleet went down on 1.7: once a node reset itself (its journal simply stops; no kernel, GPU or thermal line, the same silent reset another node showed before) and the watchdog relaunched; once every rank hit a device-side assert during a Claude Code session. Prompts with screenshots store image placeholder ids (1,000,000 and up) in the lookup's token table; a continuation ran into one and the sampled one-hot draft distribution was indexed with it. 1.7.1 stops every continuation at the first id outside the vocabulary (a CPU test reproduces the exact out-of-bounds scatter without it); wide mode comes back after a window that samples through prompts with images.

The same session answered a question that predates 1.7: why Claude Code waited about a minute on every turn. Each /v1/messages request was ~158K tokens and the prefix cache matched 4,096 of them. Two consecutive requests, captured with SGLang's runtime request dump, first differed at token 4,437: Claude Code adds a system note to the conversation on every turn (<total_tokens>… tokens left</total_tokens>), and SGLang, finding no mid-conversation system turn in V4.1's prompt format, hoists every such note into the system prompt at the top. One new line near the top moves every later token, so the whole conversation was prefilled again. The inline_system hook leaves the notes where they were, folded into the next user message; the prompt now only grows at the end.

production-1.3 → 1.5 — a crash fix, real attention heads, YOCO prefill

Three releases on 2026-09-24. 1.3 fixed a rank crash under long prompts. 1.4 stopped padding each GPU's 16 attention heads to 64 during decode and started trimming the late layers' prefill work. 1.5 skips that work outright. Each went live through a gated test window: needles, a long-prompt stress run, buried-fact checks on eight 4K–47K-token prompts, and the fixed-prompt bench.

Decode. Up to 64 query tokens, SGLang pads each rank's 16 heads to 64 because datacenter FlashMLA only has 64- and 128-head kernels. On GB10 the call goes to FlashInfer's decode_dsv4, which has 8/16/32/64/128-head builds, so the pad was four times the attention work; 16 streams (96 tokens, unpadded) ran faster than 8 (48 tokens, padded). Prefill. V4.1 builds its long-range KV only at layers 2, 8, 14 and 20. SGLang's bounded replay already ran the 22 later layers on only the last 128 tokens of each chunk, but decode reads their state only for the prompt's last 128 positions, so for every earlier chunk that pass (80 ms of a ~560 ms chunk) produced nothing anyone reads. 1.5 skips it. 1.6 (2026-09-25) adds a guard after a new crash: once, the CUDA driver refused to load a freshly compiled SGLang kernel mid-serving ("operation not permitted") and the fleet went down; a replay of the load did not reproduce it. The guard waits for the GPU and retries such a load; it has not had to act since.

production-1.2 — sixteen streams, and where a decode step goes

The request cap was 8 until 2026-09-22, far below what the KV pool allows. Decode is bound by reading the MoE weights, so a bigger batch amortizes every read: at 16 running requests the fleet delivers 1.9× the old ceiling. Capture-safe CUDA events inside the decode graphs then split a verify step into its parts, which is how the Engram stall was found and removed.

Measured — same nodes, same prompts, same day as the vLLM stack

Bring-up — seven windows