What role does memory bandwidth play in determining the actual inference cost of large language models?

Most people mistakenly assume that raw FLOPS compute power dictates inference speed, but autoregressive token generation is overwhelmingly memory bandwidth constrained. Reading model weights from High Bandwidth Memory into the processor for every generated token creates an architectural bottleneck. Understanding memory hierarchy and kernel optimization allows developers to extract five times more throughput from the identical hardware.