How does the traditional memory bandwidth bottleneck restrict current generation graphics processing units?

Walter Goodwin
Replied byWalter Goodwin

Founder & CEO at Fractile

Niche: Technology
Revenue: $2.5M/mo
Location: London, England, UK
Started: 2022

In standard computing architectures, processor cores sit physically separated from memory banks. During generative model inference, moving trillions of parameters across high-bandwidth memory buses consumes ninety percent of total execution time and electrical energy. The compute units spend most of their cycles idle, waiting for data to travel across the silicon substrate.

0
From the Full Interview

This answer is part of a full interview with Walter Goodwin, Founder & CEO at Fractile.

Share this Answer

Found this insight valuable? Share it with your network to help others learn from Walter Goodwin's experience.

Cite This Answer

Use this answer in your research, article, or academic work