Inkling-2-Small on 4× DGX Spark with SGLang (experimental)
Experimental: three days old, on a development image, with open problems listed. Inkling-2-Small (594B parameters, 17B active, the official 394 GB NVFP4 checkpoint) on four DGX Sparks with the SGLang team's own image, unmodified, plus MIT hooks for what GB10 needs: attention arguments, expert kernels, RoCE collectives, and a decode kernel that returned other requests' text as soon as two requests shared a batch. Then the speed: 61 tok/s for one stream where the image's own decode step gave 37.
4× DGX Spark GB10 · TP4/EP4 · SGLang dev-inkling-2-small · lmsysorg/sglang:dev-inkling-2-small · measured Nov 15, 2026
- Inkling-2-Small
- SGLang
- multi-node
- TP4
- DGX Spark
- FP8
- NVFP4
- MoE
01Overview
- Four DGX Spark nodes (GB10, 128 GB unified memory, arm64); one of them is rank 0, where every script runs
- A RoCE fabric between the nodes, switched or direct, with the RDMA devices visible in /sys/class/infiniband on every node
- thinkingmachines/Inkling-2-Small on the local disk of every node (394 GB each), at the revision named under Build
- About 37 GB per node for the image lmsysorg/sglang:dev-inkling-2-small
- Docker with the NVIDIA container runtime, key-based ssh from rank 0 to the other three, and permission to pass /dev/infiniband into a container
- Nothing else on the GPUs: the weights take 92–93 GB of each node and the KV pool takes what is left
thinkingmachines/Inkling-2-Small is a mixture-of-experts model with 594B parameters, 17B of them active per token. Its official checkpoint is 394 GB in NVFP4, a 4-bit floating-point weight format. This recipe serves it on four DGX Spark nodes behind one OpenAI-compatible endpoint. The model is split tensor-parallel over the four nodes (TP4/EP4), and every decode step crosses the RoCE links between them.
The engine is the SGLang team's own image for the model, lmsysorg/sglang:dev-inkling-2-small, unmodified. That image targets Hopper and datacenter Blackwell GPUs. On a Spark's GB10 it does not serve the model as it is. A set of MIT-licensed Python hooks is copied to every node and put on PYTHONPATH. The hooks supply what GB10 needs, and then the speed: about 61 tok/s for one stream, where the image's own decode step gave 37 on similar questions.
Decoding is speculative. The checkpoint ships a small draft model (the engine calls the scheme DSpark) that proposes up to 7 tokens, and the full model verifies them in one forward pass. Tool calls, reasoning with effort levels, OpenAI chat completions and responses, and Anthropic /v1/messages are served.
02Build
Run it
- Put the checkpoint on every node. It is 394 GB a node, and the launcher mounts each node's own copy. On a node that is serving something else, run the download under a memory limit (
systemd-run --user -p MemoryMax=4G hf download ...). On GB10 the page cache competes with the GPU for the same memory. - Pull the image on every node. About 37 GB, arm64. The hooks were written against the build of the tag from 2026-11-12; its digest is in the block below. If the tag has moved and a boot fails inside a hook, pull that digest and set
INKLING_IMAGEto it. - Clone the recipe on rank 0 and fill in the site file.
launch/fleet.env.exampleholds the nodes, users and RDMA devices, and every line says how to find its value. Copy it tolaunch/fleet.env. - Dry-run first.
--dry-runprints the fourdocker runcommands without starting anything. It already needs ssh to every rank, because it probes the RoCE-v2 GID index on each node. - Launch with the relaunch script. It disarms the watchdog, copies the hooks to every node, removes a previous container, starts ranks 3 to 0, waits for
/healthand arms the watchdog. A boot takes about 5 minutes with a warm kernel cache and about 17 the first time. - Read the text of overlapping requests. Run
bench/concurrency_check.pyafter every boot of a changed configuration. A broken decode path looks fast and healthy in every counter (see the deep dive).
# On every node: the checkpoint at the revision this recipe was measured on, and the image.
hf download thinkingmachines/Inkling-2-Small --revision b0e7afb8892a07897c78ce1c5b801a0d9cbd5e7b
docker pull lmsysorg/sglang:dev-inkling-2-small
# measured on lmsysorg/sglang@sha256:424c1da97f6d6eef1612006c77b2ca791657a8184c1fbdbc682ebf9485d3b32c
# On rank 0.
git clone https://github.com/aidendle94/sparkrun-recipes
cd sparkrun-recipes/recipes/inkling-2-small/sglang
cp launch/fleet.env.example launch/fleet.env # then edit it: nodes, users, RDMA devices
launch/launch-inkling2.sh --dry-run
launch/relaunch-inkling2.sh
# First request (port 8210 on rank 0).
curl -s http://<rank-0>:8210/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"inkling-2-small","messages":[{"role":"user","content":"Is 91 prime?"}],"max_tokens":400}'
# The checks to run after every boot of a changed configuration.
mkdir -p logs && python3 bench/concurrency_check.py && python3 bench/reasoning_cache_check.py --expect onThe launcher's default context is 262,144 tokens. The fleet these numbers come from serves 524,288 through the environment. That is not the default for two reasons: the 600K cap on the KV pool leaves room for one 500K conversation and little else, and --prefill-decode-interval 2 is a scheduling setting still under test.
CTX=524288 MEMFRAC=0.94 EXTRA_ARGS="--max-total-tokens 600000 --prefill-decode-interval 2" launch/relaunch-inkling2.shThe watchdog probes /health and a three-token chat completion every 60 seconds. After three failures it saves the state, logs and Python stacks of every rank and relaunches the fleet with the environment it was started in. To stop the fleet, kill the watchdog first (its pid is in the last watchdog started line of ~/inkling2-watchdog.log), then run launch/launch-inkling2.sh --stop.
03Deep dive
What GB10 needed
Nothing in the image is modified. Every change is a hook behind an environment switch, installed at interpreter start in every rank. Three of them make the model serve on GB10 at all.
- Attention. SGLang's wrapper for this GPU family dropped two arguments that Inkling passes, and the decode kernel was wrong at page size 1 (next section).
- Experts. The image's NVFP4 expert kernels are built for datacenter Blackwell only.
patches/fi_moe.pyruns the 40 NVFP4 expert layers on FlashInfer's grouped GEMM for GB10 with the checkpoint's weights. Prefill went from about 440 to about 2,000 tok/s. - Collectives. The model's all-reduces assume NVLink between GPUs in one box. Here they go over RoCE between four boxes (
patches/roce_collectives.py, a port of the vLLM adapter by Jason, @original-el8).
What makes it faster
These are all on by default, and each is switched off by setting its variable empty on the relaunch. The decode step on this hardware is set by memory bandwidth: each step reads the weights of the routed experts, so most of the levers reduce bytes read or stop one node from holding up the other three.
| Launcher variable (default) | What it does | Cost |
|---|---|---|
KV_DTYPE=fp8_e4m3 | FP8 KV cache with a reader for GB10: 1.9× the pool at the same speed | prefill NLL within 0.02 of BF16 |
GATE=0.4 | Verifies only the draft rows the draft is confident of: +15 % decode | greedy text moves at near-ties only, by less than a second request in the batch moves it |
DENSE_I8=draft,dhead,dfree | The draft reads int8 copies of its own weights: +5 % | none for the target (held-out NLL identical); about 5 % of the pool |
DENSE_P12=attn,head,free | The target's attention weights packed into 12 bits without rounding: −4.5 % step time for one stream | none: exact |
TP_SHARD=half,fp8 | Every node holds all 384 experts at a quarter of their width instead of 96 whole ones, so no node waits for the busiest: −18 % step time | not bit-identical to the stock placement (rounding only; mean NLL unchanged) |
REASONING_CACHE=1 | The server puts an answer's reasoning back into history turns that arrive without it | prompts grow by that reasoning |
TP_SHARD= DENSE_P12= is the stack as it served before 2026-11-15. KV_DTYPE=auto is the BF16 pool. NLL is the mean negative log-likelihood of held-out text, the check that a lever leaves the model alone.
The sharded experts are the largest single step. Stock expert parallelism places 96 whole experts on each node. A decode step's routed experts fall unevenly on the nodes, and every node waits in the all-reduce for the one that reads the most. With each expert cut four ways by width, every node reads the same bytes whatever the routing. The 4-bit activation scale is still taken over the full width, through one small exchange per layer, so the quantized operands are the ones the stock path computes. What differs is where the partial sums are rounded.
04Performance
Measured 2026-11-13 to 2026-11-15 (experimental: single boots on one fleet, defaults as of 2026-11-15). Four-node TP4/EP4 · lmsysorg/sglang:dev-inkling-2-small, unmodified, + MIT hooks on PYTHONPATH · DSpark drafts of up to 7 tokens with the confidence gate · FP8 KV · experts sharded by width · RoCE collectives. Decode tok/s is generated tokens over the server's own decode seconds. Image: lmsysorg/sglang:dev-inkling-2-small.
Same fleet, same questions, 2026-11-15. With the stock placement every node waits in the all-reduce for the node whose routed experts read the most bytes; sharded, every node reads the same bytes whatever the routing. Step time −18.4 % for one stream at equal accept length (−10.8 % from sharding the NVFP4 expert layers alone). Not bit-identical to the stock placement (rounding only; mean NLL unchanged).
Across recipes
Different prompt sets, all single stream at temperature 0 with completion_tokens accounting: exact within a recipe, indicative across them. Inkling-2-Small · SGLang (experimental): code questions of 400 tokens (60–61), DSpark with the confidence gate, experts sharded by width; 56.6 at temperature 0.7 · DeepSeek-V4.1-Flash · SGLang (1.8): code prompts, DSpark block 5, RoCE collectives, early Engram staging, V4.1-Flash-1017; 113.7 counting, 34.9 prose · DeepSeek-V4.1-Flash (1.1, DCP2): code prompts, DSpark k=5, b12x MoE, DCP2 (77.5 on 1.0 with a different prompt set) · GLM-5.3-Flash (1.2): structured output, thinking off · DeepSeek-V4-Flash (3.76.1): mixed prompts, 8-rep mean · DeepSeek-V4-Flash-Vision (3.73): code prompts, DCP2 + DSpark
Cold means the prompt has never been seen by the server; with prefix caching on, a repeated prompt is a cache hit and reads 3–25× higher. Inkling-2-Small · SGLang (experimental): 130K depth, cold (2,417–2,482 at ~15K) · DeepSeek-V4.1-Flash · SGLang (1.8): 32K depth, unique random words (3,848 at 128K) · DeepSeek-V4.1-Flash (1.0): 36K depth (1,940 at 132K) · GLM-5.3-Flash (1.2): 32K depth · DeepSeek-V4-Flash (3.76.1): 334K depth · DeepSeek-V4-Flash-Vision (3.73): 92K depth (2,088 at 334K)
How these were measured
Decode tok/s is generated tokens divided by decode seconds from the server's own timing, on coding questions of 400 tokens each, greedy unless the row says otherwise. Every figure is one boot between 2026-11-13 and 2026-11-15. The lab notes in the repository (docs/results.md) give the conditions and the history of each number, including the experiments that were measured and left out.
For scale: the hosted model on OpenRouter answered the same long lookups at 190–320 tok/s depending on the provider. This is a desktop cluster of four 128 GB boxes.
| Tool-call suite, 283 cases | Result |
|---|---|
| Right | 250 |
| Soft (valid calls, but not the expected ones) | 20 |
| Failed | 13: 10 are tool_choice: none, 3 are slips on the sampled trial |
| Agent loops | 12 of 12 |
Five wire formats (chat completions, Anthropic messages, each also streamed, and responses), two trials: one greedy, one with the server's default sampling. bench/toolcall_check.py.
05Notes
Using it as a client
- Send temperature 0 for lookups and agent work.A chat request without a
temperatureis sampled at 0.7. That is the server default; the model ships no generation config. - Send the reasoning back with the history.The model's template shows an earlier turn's thinking only if the client returns it (
reasoning_content, or the thinking blocks on/v1/messages). Without it, over a long context, the model answers a later question with an earlier answer. The reasoning cache covers clients that drop it on chat completions and/v1/messages. The Responses API is not covered, and a restart forgets the cache. - Give long-context turns 4,000 output tokens or more.The reasoning uses them. Treat
finish_reason: lengthwith an empty answer as no answer. - Reasoning effort is a number in the prompt, not a budget.The levels are
none,minimal,low,medium,high(the default),xhighandmax.thinking.budget_tokensis accepted and not enforced, andmax_tokensis the only hard cap. The task sets the amount of thinking more than the level does. tool_choice: nonedoes not work.The model writes the call as text. Leave the tools out of the request instead.
Known problems
- A silent scheduler hang, once.On 2026-11-13, under concurrent agent loops, the server stopped answering with every container up. It was not reproduced. The watchdog recovers from it and now captures the Python stacks.
- A short request waits behind a long prefill.It waits for the whole prefill: 338 s behind a 500K-token one.
- The reasoning loop on long lists.At 130K tokens of near-identical record lines, about 1 lookup in 10 runs out of tokens copying the list in its reasoning (0 at 60K, 0 on source code). It is the model: hosted endpoints did the same on the same questions (29 of 200 across four providers, against 4 of 40 here). No server setting fixes it. Give the model a search tool for such lists.
- The pool differs from boot to boot.493K–805K tokens. It is sized from what rank 0 has free at that moment, and the kernel's watermark boost moves that by gigabytes.
- Log-probability requests over long spans can take a node down.They materialize the full-vocabulary logits of every scored position. Keep scored spans to a few hundred tokens.
- No GPU work beside the serving fleet, however small.There is no memory for it, and the GB10 driver fails allocations instead of evicting page cache.
- Heat.Long all-node load reached 88–89 °C on the hottest node.