Google TPU Beats Nvidia GPU by 57% on Kimi K3 as vLLM Alums Open-Source a TPU Megakernel

Same model, same inference engine, same number of chips — and a 57% difference. In a head-to-head benchmark, Inferact ran Kimi K3 on 16 Google TPU v7 Ironwood chips at 709 tokens per second against 452 tokens per second on 16 Nvidia GB200s, with vLLM running on both sides and the chip plus its low-level kernel implementation as the only variables.

The result did not come from Moonshot AI, which builds Kimi, or from Google, which builds the TPU. It came from Inferact, an inference startup whose founding team is essentially the original vLLM crew. The company raised a $150 million seed round led by a16z and Lightspeed, with ZhenFund, Sequoia and Altimeter participating, at an $800 million valuation.

Part of the gain comes from DSpark, the speculative decoding framework proposed by DeepSeek: a small model guesses a run of candidate tokens, the large model verifies them in batches, and accepted guesses are skipped while rejected ones roll back. On TPU, Inferact reached an acceptance length of 6, with each decode step taking roughly 8.5 milliseconds.

Turning speculative decoding off does not erase the gap. At batch size 1 the TPU produced 249 tokens per second against the GB200's 127; at batch size 8 the TPU reached 865 against 636. On Qwen the split widens: four TPU v7 chips running Qwen 3.8 27B hit 1,515 tokens per second, versus 695 on the same configuration of GB200.

Accuracy held up. Verified with greedy decoding, Kimi K3 scored 94.4% on GPQA-Diamond and 97.2% on GSM8K on the TPU — identical to the GPU numbers.

The spec sheet cannot explain the result. The TPU v7's HBM bandwidth is 7,380 GB/s, while the GB200's is higher at 8,000 GB/s. A wider memory bus ran slower, which means the gap comes from the software layer running on top of the chips.

The technique is a megakernel: the hundreds of small kernels an inference pass normally schedules one after another are welded into a single program. All 92 MoE layers of Kimi K3 sit inside one Pallas program, so the full forward pass happens in a single call, and while layer N is still doing its MoE math, layer N+1's attention weights are already being prefetched from HBM.

TPU v7 makes that orchestration easier. Each TensorCore carries 64 MiB of software-managed VMEM, enough to hold the current layer's data and the next layer's prefetched weights at the same time, while Nvidia's Blackwell spreads roughly 38 MiB of on-chip memory across 152 SMs under automatic hardware scheduling. Writing the megakernel by hand in Pallas instead of leaning on XLA also cut compile time from more than 30 minutes to under 90 seconds.

The code is now open source in the tpu-megakernels repository, the first public result of Inferact's joint engineering work with Google Cloud, whose stated goal is to make TPU a first-class target for vLLM and to feed the optimizations back upstream. For now the megakernel is specialized for Kimi K3's architecture and needs re-tuning for other model structures, and Inferact says it plans to extend support to more architectures.