Groq's latest inference chip is betting that speed and efficiency matter more than raw memory capacity, using SRAM instead of the high-bandwidth memory (HBM) that has dominated the AI chip market. At Nvidia's GTC 2026 conference, the Groq 3 LPU drew significant attention for its unconventional approach to memory, positioning SRAM as a direct counter to HBM-equipped chips from competitors. This shift reflects a broader memory technology race emerging as the semiconductor industry hits the limits of traditional transistor scaling .

What's the Real Problem With Today's AI Memory?

For decades, the semiconductor industry relied on Moore's Law, the principle that shrinking transistor sizes would double computing power roughly every two years. However, the industry has hit a technical wall, with transistors now below 10 nanometers and further scaling becoming increasingly difficult. As a result, memory bandwidth, the speed at which data moves between memory and processors, has become the critical bottleneck limiting AI performance .

High-bandwidth memory emerged as the solution. SK Hynix pioneered HBM technology in 2013, and Samsung Electronics and SK Hynix have since dominated the market by stacking multiple layers of DRAM (dynamic RAM) to increase data throughput. However, HBM comes with significant drawbacks: it is expensive, requires complex manufacturing, and creates supply constraints that have become acute as AI demand explodes .

How Are Groq and Cerebras Challenging HBM's Dominance?

Rather than competing directly with HBM manufacturers, Groq and Cerebras Systems have chosen a different path. Both companies use SRAM (static RAM), a memory type that had attracted little attention in the AI market until recently. SRAM has critical advantages: it is faster than HBM, requires no periodic data refreshing, and reduces latency significantly. The Groq 3 LPU integrates 500 MB of SRAM directly on the chip, eliminating the communication delays that plague traditional GPU architectures .

Cerebras, which secured a $10 billion investment deal with OpenAI earlier this year, takes a similar but more ambitious approach. The company's Wafer Scale Engine (WSE) technology places both processors and memory on an uncut silicon wafer, rather than dicing the wafer into separate chips and connecting them externally. Cerebras announced in November 2024 that its product achieved speeds 75 times faster than GPUs. The company integrates 44 GB of SRAM on a single chip, delivering 7,000 times the memory bandwidth of Nvidia's H100 GPU .

The trade-off is clear: SRAM has much smaller storage capacity than HBM. The Groq 3 LPU's 500 MB pales in comparison to the 288 GB of HBM in Nvidia's Rubin GPU. Cerebras' 44 GB is larger but still falls short of HBM's capacity. To solve this limitation, Cerebras connects multiple chips together; linking four CS-3 chips yields 176 GB of total memory while maintaining the speed advantages of on-chip SRAM .

What Other Technologies Are Threatening HBM's Market Position?

Beyond SRAM-based approaches, Google has introduced a different strategy to reduce memory pressure. The company unveiled TurboQuant, a compression technology that reduces memory usage to one-sixth of previous levels. TurboQuant compresses the KV cache, temporary memory that allows AI models to remember conversation context, from 16-bit to 3-bit storage. According to Google Research, this compression could improve computational speed by up to eight times compared to Nvidia's H100 .

The impact on HBM manufacturers has been immediate. Matthew Prince, CEO of Cloudflare, compared TurboQuant to China's DeepSeek moment, when the company released a high-performance AI model for just $6 million, less than one-tenth of OpenAI's investment costs. Prince noted that TurboQuant "shows there is far more room for optimization in AI inference speed, memory usage, and power consumption." Stock prices of HBM manufacturers took a hit as investors recognized that future demand for expensive memory chips could be significantly lower than previously anticipated .

Prince

Steps to Understand the Memory Technology Landscape

• SRAM Approach: Groq and Cerebras integrate fast, low-latency memory directly on chips, sacrificing capacity for speed and efficiency in inference workloads.
• HBM Stacking: Samsung and SK Hynix stack multiple DRAM layers to increase bandwidth, currently at 12 layers with potential to exceed 16 layers, though technical constraints limit immediate implementation.
• Compression Technology: Google's TurboQuant reduces memory requirements by compressing temporary data storage, potentially decreasing demand for expensive high-bandwidth memory solutions.
• On-Chip Integration: Future HBM manufacturers may place memory directly on chips, similar to SRAM approaches, requiring fundamental changes to packaging and design processes.

Could HBM Manufacturers Fight Back?

HBM manufacturers are not sitting idle. Samsung and SK Hynix have already mastered the complex process of stacking memory layers, and some analysts forecast that placing HBM directly on-chip could materialize within two to three years. This would represent a transformative leap, combining HBM's high capacity with the speed advantages of on-chip integration .

However, such a shift requires a fundamental power reversal in the semiconductor industry. Currently, when Nvidia releases a new GPU architecture, memory manufacturers supply HBM to match its specifications. Placing memory on-chip requires an integrated design and manufacturing process from conception to final packaging. This would disrupt the current model, where Taiwan's TSMC handles packaging for most chip designers. A successful on-chip HBM strategy would require HBM manufacturers to control the entire process, a significant departure from today's supply chain structure .

The memory technology race reflects a broader shift in AI chip design. As Moore's Law falters, the industry is exploring multiple paths forward: SRAM-based inference chips optimized for speed, compression algorithms that reduce memory demands, and potential on-chip HBM solutions that could combine capacity with performance. The winner will likely be determined not by a single technology, but by which approach best balances speed, cost, and capacity for the specific workloads driving AI adoption in the coming years.