The common framing is that serving large models is a compute problem. In practice, once a model no longer fits in on-package memory, throughput is set by how much fast memory sits within reach of the accelerator, and how quickly it can be addressed.

The 2mm wall

Electrical wires on an interposer lose signal integrity quickly. Past roughly two millimetres, the energy per bit and the error rate both climb steeply, which is why HBM stacks are pressed as close to the die as packaging allows. That distance, not transistor count, is the real constraint on memory capacity per accelerator.

What changes with optics

An optical waveguide carries the same signal tens of centimetres at a fraction of the energy per bit. Moving the memory link into the optical domain decouples capacity from proximity, so an accelerator can address far more low-latency memory without a larger package.