Nvidia, IBM and Supermicro park AI memory on flash for 56x faster first tokens

The three companies announced a joint key-value cache platform on Nov. 5, 2026, aimed at large-scale AI inference. The pitch is simple: quit making GPUs recompute context they already worked through. A validated reference architecture clocked time-to-first-token speedups of as much as 56x on contexts stretching to 130,000 tokens.

Here's the problem it solves. When a model reads a prompt, it builds an internal record of that context — the KV cache. Evict it from memory and the GPU has to rebuild the whole thing on the next turn. That rebuild is called recomputation, and it eats expensive GPU cycles for zero new information.

Four pieces do the work. Supermicro's Petascale storage servers supply the physical storage layer, IBM Storage Scale manages high-performance shared storage across the system, Nvidia Dynamo runs the inference software side, and Nvidia Spectrum-X Ethernet switches shuffle data between the parts.

The numbers go well past first-token latency. Concurrent requests per second hit 4.26, up from 0.19 — a 22x jump. Total processing time for a batch of 200 requests fell by 95%. Time-to-first-token stayed nearly flat as prompt sizes grew, which is the whole point for long-context work.

The companies also ran a noisy-neighbor test, mimicking a production floor where other traffic fights for the same resources. Under that competing load, the system still held 3.6 requests per second — an 18x improvement — and chewed through 200 requests in 55.56 seconds.

The platform sits under Nvidia's Inference Context Memory Storage Platform initiative, which Nvidia introduced at CES 2026. The three vendors have been collaborating on related work since mid-2025, so this is an escalation rather than a cold open. The architecture tiers the KV cache across GPU high-bandwidth memory, DRAM, SSDs and shared flash — hot data stays close to the processor, colder context drifts outward but stays reachable without a full rebuild. The obvious targets are retrieval-augmented generation and multi-turn agents, the workloads that live and die on long context.