AidenLabrecipe 07
MEASURED · 2026-11-15
MEASURED·EXPERIMENTAL·9 min read·Nov 15, 2026

Inkling-2-Small on 4× DGX Spark with SGLang (experimental)

Experimental: three days old, on a development image, with open problems listed. Inkling-2-Small (594B parameters, 17B active, the official 394 GB NVFP4 checkpoint) on four DGX Sparks with the SGLang team's own image, unmodified, plus MIT hooks for what GB10 needs: attention arguments, expert kernels, RoCE collectives, and a decode kernel that returned other requests' text as soon as two requests shared a batch. Then the speed: 61 tok/s for one stream where the image's own decode step gave 37.

4× DGX Spark GB10 · TP4/EP4 · SGLang dev-inkling-2-small · lmsysorg/sglang:dev-inkling-2-small · measured Nov 15, 2026

  • Inkling-2-Small
  • SGLang
  • multi-node
  • TP4
  • DGX Spark
  • FP8
  • NVFP4
  • MoE
2,417–2,482tok/sPrefill · ~15K-token promptscold 130K / 260K / 500K tokens in 61 / 142 / 348 s
60–61tok/sDecode · one stream, greedycoding questions, 400 tokens each
262KtokensContextlauncher default
493K–805KtokensKV poolFP8, sized at each boot from what rank 0 has free

01Overview

What you need
  • Four DGX Spark nodes (GB10, 128 GB unified memory, arm64); one of them is rank 0, where every script runs
  • A RoCE fabric between the nodes, switched or direct, with the RDMA devices visible in /sys/class/infiniband on every node
  • thinkingmachines/Inkling-2-Small on the local disk of every node (394 GB each), at the revision named under Build
  • About 37 GB per node for the image lmsysorg/sglang:dev-inkling-2-small
  • Docker with the NVIDIA container runtime, key-based ssh from rank 0 to the other three, and permission to pass /dev/infiniband into a container
  • Nothing else on the GPUs: the weights take 92–93 GB of each node and the KV pool takes what is left

thinkingmachines/Inkling-2-Small is a mixture-of-experts model with 594B parameters, 17B of them active per token. Its official checkpoint is 394 GB in NVFP4, a 4-bit floating-point weight format. This recipe serves it on four DGX Spark nodes behind one OpenAI-compatible endpoint. The model is split tensor-parallel over the four nodes (TP4/EP4), and every decode step crosses the RoCE links between them.

The engine is the SGLang team's own image for the model, lmsysorg/sglang:dev-inkling-2-small, unmodified. That image targets Hopper and datacenter Blackwell GPUs. On a Spark's GB10 it does not serve the model as it is. A set of MIT-licensed Python hooks is copied to every node and put on PYTHONPATH. The hooks supply what GB10 needs, and then the speed: about 61 tok/s for one stream, where the image's own decode step gave 37 on similar questions.

Decoding is speculative. The checkpoint ships a small draft model (the engine calls the scheme DSpark) that proposes up to 7 tokens, and the full model verifies them in one forward pass. Tool calls, reasoning with effort levels, OpenAI chat completions and responses, and Anthropic /v1/messages are served.

02Build

Run it

  1. Put the checkpoint on every node. It is 394 GB a node, and the launcher mounts each node's own copy. On a node that is serving something else, run the download under a memory limit (systemd-run --user -p MemoryMax=4G hf download ...). On GB10 the page cache competes with the GPU for the same memory.
  2. Pull the image on every node. About 37 GB, arm64. The hooks were written against the build of the tag from 2026-11-12; its digest is in the block below. If the tag has moved and a boot fails inside a hook, pull that digest and set INKLING_IMAGE to it.
  3. Clone the recipe on rank 0 and fill in the site file. launch/fleet.env.example holds the nodes, users and RDMA devices, and every line says how to find its value. Copy it to launch/fleet.env.
  4. Dry-run first. --dry-run prints the four docker run commands without starting anything. It already needs ssh to every rank, because it probes the RoCE-v2 GID index on each node.
  5. Launch with the relaunch script. It disarms the watchdog, copies the hooks to every node, removes a previous container, starts ranks 3 to 0, waits for /health and arms the watchdog. A boot takes about 5 minutes with a warm kernel cache and about 17 the first time.
  6. Read the text of overlapping requests. Run bench/concurrency_check.py after every boot of a changed configuration. A broken decode path looks fast and healthy in every counter (see the deep dive).
launch · rank 0
# On every node: the checkpoint at the revision this recipe was measured on, and the image.
hf download thinkingmachines/Inkling-2-Small --revision b0e7afb8892a07897c78ce1c5b801a0d9cbd5e7b
docker pull lmsysorg/sglang:dev-inkling-2-small
#   measured on lmsysorg/sglang@sha256:424c1da97f6d6eef1612006c77b2ca791657a8184c1fbdbc682ebf9485d3b32c

# On rank 0.
git clone https://github.com/aidendle94/sparkrun-recipes
cd sparkrun-recipes/recipes/inkling-2-small/sglang
cp launch/fleet.env.example launch/fleet.env      # then edit it: nodes, users, RDMA devices
launch/launch-inkling2.sh --dry-run
launch/relaunch-inkling2.sh

# First request (port 8210 on rank 0).
curl -s http://<rank-0>:8210/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"inkling-2-small","messages":[{"role":"user","content":"Is 91 prime?"}],"max_tokens":400}'

# The checks to run after every boot of a changed configuration.
mkdir -p logs && python3 bench/concurrency_check.py && python3 bench/reasoning_cache_check.py --expect on

The launcher's default context is 262,144 tokens. The fleet these numbers come from serves 524,288 through the environment. That is not the default for two reasons: the 600K cap on the KV pool leaves room for one 500K conversation and little else, and --prefill-decode-interval 2 is a scheduling setting still under test.

the 500K configuration
CTX=524288 MEMFRAC=0.94 EXTRA_ARGS="--max-total-tokens 600000 --prefill-decode-interval 2" launch/relaunch-inkling2.sh

The watchdog probes /health and a three-token chat completion every 60 seconds. After three failures it saves the state, logs and Python stacks of every rank and relaunches the fleet with the environment it was started in. To stop the fleet, kill the watchdog first (its pid is in the last watchdog started line of ~/inkling2-watchdog.log), then run launch/launch-inkling2.sh --stop.

03Deep dive

What GB10 needed

Nothing in the image is modified. Every change is a hook behind an environment switch, installed at interpreter start in every rank. Three of them make the model serve on GB10 at all.

  • Attention. SGLang's wrapper for this GPU family dropped two arguments that Inkling passes, and the decode kernel was wrong at page size 1 (next section).
  • Experts. The image's NVFP4 expert kernels are built for datacenter Blackwell only. patches/fi_moe.py runs the 40 NVFP4 expert layers on FlashInfer's grouped GEMM for GB10 with the checkpoint's weights. Prefill went from about 440 to about 2,000 tok/s.
  • Collectives. The model's all-reduces assume NVLink between GPUs in one box. Here they go over RoCE between four boxes (patches/roce_collectives.py, a port of the vLLM adapter by Jason, @original-el8).

What makes it faster

These are all on by default, and each is switched off by setting its variable empty on the relaunch. The decode step on this hardware is set by memory bandwidth: each step reads the weights of the routed experts, so most of the levers reduce bytes read or stop one node from holding up the other three.

Launcher variable (default)What it doesCost
KV_DTYPE=fp8_e4m3FP8 KV cache with a reader for GB10: 1.9× the pool at the same speedprefill NLL within 0.02 of BF16
GATE=0.4Verifies only the draft rows the draft is confident of: +15 % decodegreedy text moves at near-ties only, by less than a second request in the batch moves it
DENSE_I8=draft,dhead,dfreeThe draft reads int8 copies of its own weights: +5 %none for the target (held-out NLL identical); about 5 % of the pool
DENSE_P12=attn,head,freeThe target's attention weights packed into 12 bits without rounding: −4.5 % step time for one streamnone: exact
TP_SHARD=half,fp8Every node holds all 384 experts at a quarter of their width instead of 96 whole ones, so no node waits for the busiest: −18 % step timenot bit-identical to the stock placement (rounding only; mean NLL unchanged)
REASONING_CACHE=1The server puts an answer's reasoning back into history turns that arrive without itprompts grow by that reasoning

TP_SHARD= DENSE_P12= is the stack as it served before 2026-11-15. KV_DTYPE=auto is the BF16 pool. NLL is the mean negative log-likelihood of held-out text, the check that a lever leaves the model alone.

The sharded experts are the largest single step. Stock expert parallelism places 96 whole experts on each node. A decode step's routed experts fall unevenly on the nodes, and every node waits in the all-reduce for the one that reads the most. With each expert cut four ways by width, every node reads the same bytes whatever the routing. The 4-bit activation scale is still taken over the full width, through one small exchange per layer, so the quantized operands are the ones the stock path computes. What differs is where the partial sums are rounded.

04Performance

Measured 2026-11-13 to 2026-11-15 (experimental: single boots on one fleet, defaults as of 2026-11-15). Four-node TP4/EP4 · lmsysorg/sglang:dev-inkling-2-small, unmodified, + MIT hooks on PYTHONPATH · DSpark drafts of up to 7 tokens with the confidence gate · FP8 KV · experts sharded by width · RoCE collectives. Decode tok/s is generated tokens over the server's own decode seconds. Image: lmsysorg/sglang:dev-inkling-2-small.

Prefill · ~15K-token prompts2,417–2,482tok/scold 130K / 260K / 500K tokens in 61 / 142 / 348 s; 2,337–2,409 with the stock expert placement
Decode · one stream, greedy60–61tok/scoding questions, 400 tokens each; 56.6 at temperature 0.7; 37 with the image's own decode step on another set of such questions
Context262Ktokenslauncher default; 524,288 run through the environment with the pool capped at 600K
KV pool493K–805KtokensFP8, sized at each boot from what rank 0 has free; 545K–600K with the 600K cap
Aggregate · 4 / 8 streams106 / 130tok/sgreedy, tokens of the batch per decode second; about 26 / 16 a stream; cap 8 running requests
Decode behind a long context52–55tok/sone stream after 130K–500K tokens of context
Tool calls250 / 283right20 soft, 13 failed (10 are `tool_choice: none`, 3 slips on the sampled trial); 12 of 12 agent loops
Boot to healthy~5minwarm kernel cache; about 17 min the first time
Hooks16filesMIT; the base image is unmodified, every change sits behind an environment switch
The stack before and after 2026-11-15: experts sharded by width, attention weights packedtok/s, greedy, 400-token coding answers · higher is better
96 whole experts a node (stock placement)all 384 experts at quarter width + 12-bit attention weights
1 stream48.360.0+24%
4 streams, batch total88.0106+20%
8 streams, batch total114130+14%

Same fleet, same questions, 2026-11-15. With the stock placement every node waits in the all-reduce for the node whose routed experts read the most bytes; sharded, every node reads the same bytes whatever the routing. Step time −18.4 % for one stream at equal accept length (−10.8 % from sharding the NVFP4 expert layers alone). Not bit-identical to the stock placement (rounding only; mean NLL unchanged).

Across recipes

Single-stream decode, by recipetok/s · higher is better
Inkling-2-Small · SGLang (experimental)shipped60.0
DeepSeek-V4.1-Flash · SGLang (1.8)96.5
DeepSeek-V4.1-Flash (1.1, DCP2)60.8
GLM-5.3-Flash (1.2)89.7
DeepSeek-V4-Flash (3.76.1)71.0
DeepSeek-V4-Flash-Vision (3.73)67.9

Different prompt sets, all single stream at temperature 0 with completion_tokens accounting: exact within a recipe, indicative across them. Inkling-2-Small · SGLang (experimental): code questions of 400 tokens (60–61), DSpark with the confidence gate, experts sharded by width; 56.6 at temperature 0.7 · DeepSeek-V4.1-Flash · SGLang (1.8): code prompts, DSpark block 5, RoCE collectives, early Engram staging, V4.1-Flash-1017; 113.7 counting, 34.9 prose · DeepSeek-V4.1-Flash (1.1, DCP2): code prompts, DSpark k=5, b12x MoE, DCP2 (77.5 on 1.0 with a different prompt set) · GLM-5.3-Flash (1.2): structured output, thinking off · DeepSeek-V4-Flash (3.76.1): mixed prompts, 8-rep mean · DeepSeek-V4-Flash-Vision (3.73): code prompts, DCP2 + DSpark

Cold prefill, by recipetok/s · unique content · higher is better
Inkling-2-Small · SGLang (experimental)shipped2,096
DeepSeek-V4.1-Flash · SGLang (1.8)4,892
DeepSeek-V4.1-Flash (1.0)2,143
GLM-5.3-Flash (1.2)2,070
DeepSeek-V4-Flash (3.76.1)2,200
DeepSeek-V4-Flash-Vision (3.73)2,380

Cold means the prompt has never been seen by the server; with prefix caching on, a repeated prompt is a cache hit and reads 3–25× higher. Inkling-2-Small · SGLang (experimental): 130K depth, cold (2,417–2,482 at ~15K) · DeepSeek-V4.1-Flash · SGLang (1.8): 32K depth, unique random words (3,848 at 128K) · DeepSeek-V4.1-Flash (1.0): 36K depth (1,940 at 132K) · GLM-5.3-Flash (1.2): 32K depth · DeepSeek-V4-Flash (3.76.1): 334K depth · DeepSeek-V4-Flash-Vision (3.73): 92K depth (2,088 at 334K)

How these were measured

Decode tok/s is generated tokens divided by decode seconds from the server's own timing, on coding questions of 400 tokens each, greedy unless the row says otherwise. Every figure is one boot between 2026-11-13 and 2026-11-15. The lab notes in the repository (docs/results.md) give the conditions and the history of each number, including the experiments that were measured and left out.

For scale: the hosted model on OpenRouter answered the same long lookups at 190–320 tok/s depending on the provider. This is a desktop cluster of four 128 GB boxes.

Tool-call suite, 283 casesResult
Right250
Soft (valid calls, but not the expected ones)20
Failed13: 10 are tool_choice: none, 3 are slips on the sampled trial
Agent loops12 of 12

Five wire formats (chat completions, Anthropic messages, each also streamed, and responses), two trials: one greedy, one with the server's default sampling. bench/toolcall_check.py.

05Notes

Using it as a client

  • Send temperature 0 for lookups and agent work.A chat request without a temperature is sampled at 0.7. That is the server default; the model ships no generation config.
  • Send the reasoning back with the history.The model's template shows an earlier turn's thinking only if the client returns it (reasoning_content, or the thinking blocks on /v1/messages). Without it, over a long context, the model answers a later question with an earlier answer. The reasoning cache covers clients that drop it on chat completions and /v1/messages. The Responses API is not covered, and a restart forgets the cache.
  • Give long-context turns 4,000 output tokens or more.The reasoning uses them. Treat finish_reason: length with an empty answer as no answer.
  • Reasoning effort is a number in the prompt, not a budget.The levels are none, minimal, low, medium, high (the default), xhigh and max. thinking.budget_tokens is accepted and not enforced, and max_tokens is the only hard cap. The task sets the amount of thinking more than the level does.
  • tool_choice: none does not work.The model writes the call as text. Leave the tools out of the request instead.

Known problems

  • A silent scheduler hang, once.On 2026-11-13, under concurrent agent loops, the server stopped answering with every container up. It was not reproduced. The watchdog recovers from it and now captures the Python stacks.
  • A short request waits behind a long prefill.It waits for the whole prefill: 338 s behind a 500K-token one.
  • The reasoning loop on long lists.At 130K tokens of near-identical record lines, about 1 lookup in 10 runs out of tokens copying the list in its reasoning (0 at 60K, 0 on source code). It is the model: hosted endpoints did the same on the same questions (29 of 200 across four providers, against 4 of 40 here). No server setting fixes it. Give the model a search tool for such lists.
  • The pool differs from boot to boot.493K–805K tokens. It is sized from what rank 0 has free at that moment, and the kernel's watermark boost moves that by gigabytes.
  • Log-probability requests over long spans can take a node down.They materialize the full-vocabulary logits of every scored position. Keep scored spans to a few hundred tokens.
  • No GPU work beside the serving fleet, however small.There is no memory for it, and the GB10 driver fails allocations instead of evicting page cache.
  • Heat.Long all-node load reached 88–89 °C on the hottest node.
© 2026 AidenLabBuilt on measured results, not vendor claims.