Hand the memory to the CPU: Intel lays out a data-center KV cache strategy

Every token a large model generates requires it to look back at what came before, and inference systems cache the intermediate results so the work is not repeated. That cache, the KV cache, grows with the length of a session and with how much context an agent reads. Handing that memory to the CPU instead of leaving all of it in GPU memory is the premise behind Intel's approach to data-center inference.

The pressure is easiest to see in agent workloads. A coding agent debugging across a dozen files keeps reading code, searching references and running tests, and the accumulated history compounds. When GPU memory fills up, cache blocks get evicted; when the agent needs that history again, the system may have to redo prefill, and the time to first token climbs. In a rough calculation published by QbitAI on September 16, Qwen3-8B carries about 147KB of KV data per token at BF16 or FP16, which would come to roughly 147GB for a one-million-token context. These are illustrative assumptions rather than measurements of any real service, but they show why the bill grows so quickly.

Intel's answer is hierarchical offloading: keep actively used cache in high-bandwidth GPU memory, move cache likely to be reused soon into CPU-side DDR memory, and push colder but still valuable history down to SSD or remote storage. In that design the CPU manages inference memory rather than sitting idle beside it.

Four directions make up the strategy. KV Shrink combines tiered management with compression and exposes hot and cold scheduling APIs; KV Fuse tackles the difficulty of reusing caches generated separately from different documents; KV Cascade has a helper model pre-filter context so the main model handles less; KV Infinity targets continuously growing long tasks with on-demand loading and prefetching. KV Shrink is the one with the most concrete implementation and test data disclosed so far.

Compression is where Intel's silicon comes in. Its QAT, or QuickAssist Technology, accelerator, integrated from certain fourth-generation Xeon Scalable processors onward, takes over compression and decompression so general-purpose CPU cores stay free. Intel also rearranged the KV cache storage format so the data compresses more effectively, lifting the space saved from more than 10% to more than 20%, for an overall reduction of about 20% to 30%. The compression is lossless, so decompression restores the original KV data.

The speed comparisons are the more revealing part. In one test using dual Xeon Gold 6554S processors, two Nvidia L20 GPUs and a Qwen3-32B model at an 80% cache hit rate, KV Shrink delivered up to about 5x faster time to first token than a native vLLM baseline without tiered offloading across the input lengths and concurrency combinations tested. QAT hardware compression ran at roughly twice the performance of a CPU software compression path, and enabling compression added less than 10% of TTFT overhead.

A second test, run with the DaoCloud joint lab, is closer to a coding agent service. On dual Xeon Gold 6554S processors with eight H800 GPUs and a Qwen3-32B FP8 model, again at an 80% hit rate, KV Shrink cut average TTFT from 129.81 milliseconds to 114.13 milliseconds under single-stream load, about 12.1% lower, and about 4.6% lower at eight-way concurrency, compared with the LMCache setup used in the test. The article is careful to note that changing the hardware or the comparison target changes the gain, which is why operators have to test with their own context lengths, concurrency and hit rates.

Intel is not the only company working here: vLLM's KV offloading already supports moving cache blocks to CPU memory with configurable secondary storage tiers, so Intel's differentiation rests on QAT hardware compression and the storage-format rearrangement. It will hold its Intel Technology Innovation and Industry Ecosystem Conference on September 22-23. For data-center operators, the question that matters is not the architecture diagram but the arithmetic: can the same fleet of GPUs serve more requests at acceptable latency and cost.