Markdown edition

# vLLM’s Kimi K3 decode path reaches 842 tokens a second in a specific GB300 test

> vLLM reports 842 versus 488 aggregate decode tokens a second for Kimi K3 on 16 GB300 GPUs under a specific DSpark setup. Prefill is excluded.

By BIG CHANGE Editorial

Published: 2026-10-18T20:03:34.885Z
Updated: 2026-10-18T20:03:34.885Z
Canonical: https://bigchange.ai/blog/kimi-k3-vllm-gpu-inference-benchmark

![Conceptual charcoal illustration of the rear of a generic compute cabinet with rack units and secured cables in a data-center aisle.](https://bigchange.ai/api/media/file/kimi-k3-decode-cabinet-concept-v1.png)
AI-generated conceptual illustration by BIG CHANGE; no actual GB300 chassis, vLLM installation or measured result is depicted.

vLLM has merged an optional, model-specific decode path for Kimi K3 on NVIDIA Blackwell GPUs. In its [October 16 account](https://vllm.ai/blog/2026-10-16-kimi-k3-megakernel), the project reports 842 generated tokens a second against 488 for stock vLLM, a 1.73× gain. That comparison has a narrow boundary: one request at a time, tensor parallelism across 16 GB300 GPUs, DSpark speculative decoding with seven draft tokens, a synthetic acceptance length of six, a one-token input and a 1,000-token output. The figure measures aggregate decoding throughput after prefill. It does not measure how fast an entire user request finishes.

For engineers deciding whether to evaluate this path, the useful development is the combination of a substantial decode result and a fairly explicit account of where it applies. The [merged implementation](https://github.com/vllm-project/vllm/pull/61867) is opt-in through `decode_backend: latency_specialized`; the default `auto` setting continues to use stock decoding. vLLM says the specialized path targets GB300 deployments with eight or 16 GPUs in one NVLink domain, while prefills, mixed batches and unsupported steps fall back to the standard model forward path.

## Where the time went

Kimi K3 is a large mixture-of-experts model, according to its [model card](https://huggingface.co/moonshotai/Kimi-K3). During decoding, a serving system repeatedly runs attention, chooses experts and exchanges data among GPUs to produce the next tokens. The vLLM team built a CuTe DSL path that specializes those operations for the shapes it expects. Its account describes different attention projection kernels for small and larger batches; fusion of the attention output projection, cross-GPU reduction and following residual work; sharded router calculations; overlap between shared and routed experts; and prefetching weights during communication. In a per-layer trace shown in the post, the specialized path uses 13 kernels and 69 microseconds where stock vLLM uses 23 kernels and 130 microseconds. This is a reported trace for the illustrated setup, not an application latency figure.

The scope matters as much as the kernel design. The [pull request](https://github.com/vllm-project/vllm/pull/61867) describes GB300 validation at TP8 and TP16, non-speculative decode up to batch 64, DSpark verification with three to seven draft tokens, and optional decode context parallelism. It states that the performance work used FP8 KV cache, FP32 SSM cache and model-provider weights. The path leaves model weights, scheduling, sampling and the DSpark drafter unchanged. That lets a serving team compare a changed decode implementation with a familiar surrounding stack, but it also means the optimization does not speed every part of serving.

## Read the two benchmark tables separately

The 842-versus-488 result uses **TP16 on 16 GB300 GPUs, batch 1, DSpark with seven draft tokens, synthetic acceptance length six, DL=7, ISL=1 and OSL=1k**. vLLM calls the metric aggregate decode tokens per second and excludes prefill. Synthetic acceptance fixes how many proposed tokens the benchmark accepts; it is not a measured acceptance rate from users’ prompts. A deployment with different prompts, acceptance behavior, batch sizes or input lengths needs its own measurement.

The post also gives a [non-speculative table](https://vllm.ai/blog/2026-10-16-kimi-k3-megakernel#decode-performance-without-speculation). At TP16, for example, it reports 256.4 tokens a second at batch 1 and 3,606.1 at batch 64, with gains of 1.88× and 1.62× over the respective stock configurations. Those are aggregate TP-group results without speculative decoding. They cannot be combined with the DSpark number to claim a faster per-user experience at batch 64.

Another number in the post concerns [AgentX](https://vllm.ai/blog/2026-10-16-kimi-k3-megakernel#improvement-on-agentx), a workload that replays coding-agent traffic patterns with synthetic content. vLLM reports P90 interactivity of 403.2 versus 223.7 tokens per second per user at concurrency one. The optimized result ran on **16 GB300 GPUs**, while the earlier comparison ran on **eight**. Because GPU count changes along with the implementation, that 1.8× comparison does not isolate the decode-path improvement. AgentX also includes longer contexts, turns, pauses and cache reuse, so its metric answers a different question from the pure decode benchmark.

There is a small discrepancy between the public post and the earlier [pull-request performance table](https://github.com/vllm-project/vllm/pull/61867): the latter lists 838 versus 489 tokens a second for TP16, batch 1, DSpark DL7 and synthetic acceptance six. The post’s 842 versus 488 comparison is the source of the 1.73× figure here. The earlier table is useful evidence of a similar run, but the two pairs should not be spliced together. vLLM has published the code and its own validation; BIG CHANGE has not reproduced either run.

## What an evaluation would need

A meaningful local comparison would start with the [merged PR and its stated configuration](https://github.com/vllm-project/vllm/pull/61867): the same Kimi K3 model-provider weights, a GB300 system with 16 GPUs in one NVLink domain, TP16, the cache formats and the exact DSpark setup for the headline test. Run stock `auto` and opt-in `latency_specialized` under the same workload, then record decode throughput separately from prefill, time to first token, output throughput and request-level latency. A workstation or a smaller GPU pool cannot directly reproduce the 16-GB300 headline. The post does not establish a rental cost for that setup.

The [PR’s validation account](https://github.com/vllm-project/vllm/pull/61867) reports numerical checks and model evaluations including AIME25, GPQA-Diamond and a long-context MRCR test. It also describes a synchronization error found during development and variation in long-output tests. Those checks give readers more to inspect than a speed chart, while remaining the implementers’ evidence. The decision for a serving team is whether its actual hardware, batching, context lengths and speculative acceptance resemble the supported configurations closely enough to justify measuring this opt-in path on its own traffic.

## The big change

vLLM’s merged Kimi K3 path makes a specialized, optional GPU decode implementation available with a documented benchmark and validation trail. The strongest public headline is **842 versus 488 aggregate decode tokens a second, or 1.73×, for TP16 on 16 GB300 GPUs at batch 1 with DSpark’s seven draft tokens and synthetic acceptance length six**. Prefill is excluded. Whether it improves a particular service depends on that service’s hardware and complete request workload.

## Sources & further reading

- [vLLM’s October 16 technical post](https://vllm.ai/blog/2026-10-16-kimi-k3-megakernel) explains the decode changes and supplies the 842/488 setup, the non-speculative table and the AgentX comparison. Its results are reported by the project and NVIDIA contributor Benjamin Chislett.
- [Merged vLLM pull request #61867](https://github.com/vllm-project/vllm/pull/61867) documents the opt-in setting, supported configurations, implementation boundary, validation and an earlier 838/489 DSpark table. It is the code and results trail, not independent replication.
- [Moonshot AI’s Kimi K3 model card](https://huggingface.co/moonshotai/Kimi-K3) identifies the model and weights referenced by the implementation.
- [AgentX methodology](https://inferencex.semianalysis.com/agentx) describes the traffic-replay workload behind the separate interactivity comparison.

## Sources

- [vLLM, Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly (2026-10-16)](https://vllm.ai/blog/2026-10-16-kimi-k3-megakernel) — First-party technical account and source of the 842/488, 1.73× result. It specifies TP16 on 16 GB300 GPUs, batch 1, DSpark DL7, synthetic AL6, ISL1, OSL1k and aggregate decode TPS excluding prefill. Also supplies distinct non-speculative and AgentX results.
- [vLLM pull request #61867, Kimi K3 specialized decode](https://github.com/vllm-project/vllm/pull/61867) — Merged implementation and author-run validation record. Documents the final opt-in setting, supported configurations and fallback. Its earlier DSpark table says 838/489 at TP16 B1 synthetic AL6, so those figures are not combined with the blog pair.
- [Moonshot AI, Kimi K3 model card](https://huggingface.co/moonshotai/Kimi-K3) — Identifies the model and model-provider weights; it does not independently establish vLLM performance.
- [InferenceX, AgentX Methodology and Datasets](https://inferencex.semianalysis.com/agentx) — Describes the traffic-replay workload used for the separately reported interactivity comparison. The vLLM comparison also changes GPU count.