HBM Memory and the Real Limits of Modern AI Accelerators

AI accelerator specs are almost always led by compute figures. In real inference workloads, the number that matters more sits one line below: memory bandwidth.

Arithmetic intensity, and why it matters

Every compute operation needs data moved from memory. The ratio of operations to bytes moved is called arithmetic intensity, and that ratio determines whether a workload is compute-bound or memory-bound.

Model training usually processes data in large batches, so the same weights are used for many samples at once — intensity is high, and compute becomes the deciding factor. Inference that produces one token at a time sits at the opposite end: every layer's weights must be read to produce a single token.

A simple and surprising calculation

Token generation speed on a large model can be estimated without knowing the compute figure at all. Divide memory bandwidth by model weight size, and the result approaches the upper bound of tokens per second.

A 140 GB model on an accelerator with 3 TB per second of bandwidth yields roughly twenty tokens per second as a theoretical ceiling. Doubling compute does not change that number at all; doubling memory bandwidth doubles it.

How HBM answers this

High Bandwidth Memory takes a different approach from conventional graphics memory. Instead of raising clock speeds on a narrow bus, HBM stacks several memory dies vertically and connects them through thousands of very wide lanes, placed right next to the processor on the same substrate.

Short lanes in huge numbers deliver large bandwidth at low frequencies, which also holds down power consumption per byte. The price is a complicated manufacturing process and far higher cost, plus capacity limited by the physical space around the processor.

How to read specs correctly

For inference workloads, the two most useful numbers are memory capacity and memory bandwidth. Capacity determines whether the model fits at all; bandwidth determines how fast it runs once it fits.

Compute figures are the deciding factor in training, large-batch processing, and the long-context prefill stage. Reading specs without knowing which workload will run almost always produces a suboptimal choice.

Where the industry is heading

Because the gap is structural, most innovation is aimed at reducing the need to move data: quantization that shrinks weights, architectures that activate only some parameters for each token, and restructuring computation so data already in cache is used as much as possible.

All of them attack the same problem from the software side, because adding bandwidth in hardware is far more expensive than adding compute.