SambaNova says Supermicro plans to build and sell inference systems combining NVIDIA GPUs with its SN50 chips. The initial plan covers 10,000 SN50 chips.
The big change
- Splitting phases can match hardware to different compute and memory demands.
- Supermicro would deliver and support the systems; SambaNova would tune models.
- The chip count is a plan, not confirmed shipments.
Why split inference?
A language model processes a prompt before generating response tokens. The Splitwise research paper found prompt processing compute-intensive and token generation limited by memory under its studied conditions. Separating them lets operators allocate hardware suited to each phase while running one service.
In the November 3 announcement, the partners propose NVIDIA GPUs for prompt processing and SambaNova’s dataflow chips for generation. This does not prove that their rack is faster or more efficient.
SambaNova’s earlier architecture account describes on-chip memory and dataflow execution in the SN50. It is a vendor description, not a measurement of the proposed combined rack.
Splitting stages requires transferring request state. Splitwise studied transfer overhead in its own system; it did not measure this proposed configuration. The partners have not supplied comparable independent results.
The companies plan to start with neoclouds and sovereign clouds, then enterprises. Prices, delivery dates and confirmed customer deployments were not disclosed.
