NVIDIA's new Groq 3 Language Processing Unit (LPU) is a specialized chip designed exclusively for one task: generating text tokens from large language models at dramatically lower power consumption than traditional GPUs. At GTC 2026, NVIDIA announced a $20 billion licensing deal with Groq and integrated the LPU into its Vera Rubin platform, marking a strategic shift away from general-purpose computing toward task-specific optimization. The chip achieves roughly 150 tokens per watt for a 70-billion-parameter model, compared to about 4.3 tokens per watt for NVIDIA's H100 GPU, a 35-fold improvement in inference efficiency .

What Makes the Groq 3 LPU Different From a GPU?

The fundamental difference lies in memory architecture. Traditional GPUs like the H100 rely on off-chip HBM (high-bandwidth memory) with roughly 3.35 terabytes per second of bandwidth. The Groq 3 LPU replaces this entirely with 500 megabytes of on-chip SRAM (static random-access memory) sitting directly on the processor die, delivering 150 terabytes per second of memory bandwidth per chip. To put this in perspective, that is roughly 45 times more memory bandwidth than an H100 .

This architectural choice makes sense for a very specific workload: the decode phase of language model inference. When a model generates text, it reads the entire weight matrix and key-value cache from memory for each token produced. On a GPU, this memory access becomes the bottleneck. For a 70-billion-parameter model at FP8 precision (a common compressed format), reading 70 gigabytes of weights takes about 21 milliseconds on an H100. On the Groq 3 LPU, the same operation takes approximately 0.47 milliseconds because the data never leaves the chip .

The tradeoff is inflexibility. The LPU cannot train models, cannot run the prefill phase (where the model processes the input prompt all at once), and has no support for computer vision or video generation. It does one thing exceptionally well: autoregressive token generation for dense language model serving .

How Does the LPU Fit Into NVIDIA's Broader Strategy?

Rather than positioning the LPU as a GPU replacement, NVIDIA designed it as a complementary component within the Vera Rubin platform. The Vera Rubin system integrates multiple specialized chips: the Vera CPU handles system-level scheduling and task management, the Rubin GPU manages prefill operations and attention computations, and the Groq 3 LPU handles the decode phase where tokens are generated sequentially .

This disaggregated architecture reflects a broader industry shift. Jensen Huang, NVIDIA's CEO, explained that the focus is no longer on a single processor but on a complete computing system built around the concept of an "AI Factory." When people mentioned Hopper in the past, they thought of a single chip; when mentioning Vera Rubin, it refers to an entire system optimized for different stages of AI workloads .

The Vera Rubin NVL72 rack system, which integrates 72 Rubin GPUs and 36 Vera CPUs, demonstrates this philosophy. Compared with the previous-generation Blackwell platform, the system requires only one-quarter of the GPUs for training large hybrid-expert models, while inference throughput per watt can increase by approximately 10 times .

What Are the Practical Implications for Data Center Operators?

For companies running millions of inference requests per day, the economics matter significantly. An H100 drawing 700 watts and generating 3,000 tokens per second for 70-billion-parameter model serving delivers roughly 4.3 tokens per watt. The Groq 3 LPU achieves around 150 tokens per watt for comparable model sizes, representing a 35-fold improvement in inference efficiency per watt .

However, the LPU has capacity constraints. Each chip contains 500 megabytes of on-chip SRAM. A single Groq 3 LPX rack contains 256 LPU chips, providing 128 gigabytes of aggregate on-chip SRAM. A 70-billion-parameter model at FP8 precision fits inside a single LPX rack with room for the key-value cache, but larger models cannot be deployed on a single rack without additional hardware .

How to Evaluate Whether the Groq 3 LPU Fits Your Infrastructure

• Workload Profile: The LPU is optimal only if your inference workload is decode-heavy and latency-sensitive. If you run significant prefill operations, fine-tuning, or training, you still need GPUs as your primary compute resource.
• Model Size and Precision: Verify that your model fits within the on-chip SRAM capacity of your available LPU system. A 70-billion-parameter model at FP8 fits in a single LPX rack, but larger models or higher precision formats may require multiple racks or GPU-based alternatives.
• Power Budget and Cost: Calculate your total inference cost by multiplying tokens generated per watt by your electricity rate and hardware amortization. The 35-fold efficiency gain translates to significant operational savings only if decode throughput is your primary cost driver.
• Deterministic Latency Requirements: The LPU offers flat decode latency regardless of batch size variation, unlike GPUs which experience variable latency due to cache misses and memory controller queuing. If your application requires predictable response times, this architectural advantage may justify the inflexibility.

What Does Deterministic Execution Mean for Real-Time AI?

The LPU uses Groq's spatial execution model, where the compiler orchestrates computation, data transfer, and synchronization in advance rather than relying on runtime dynamic scheduling. This means each clock cycle executes exactly the same operations in the same order, with no prefetching logic, cache hierarchy, or speculative execution .

For real-time inference applications, this deterministic execution stabilizes first-token latency and per-token generation time, maintaining consistent response even with small batch sizes. Memory access and compute are precisely aligned, instruction timing is explicitly controlled by the compiler, and performance variation under different workloads is minimized. This predictability is difficult to achieve on GPUs, where memory access patterns and cache behavior introduce variability .

The LPU integrates high-speed chip-to-chip interconnects to enable distributed inference pipelines. Each LPU features 96 links running at 112 gigabits per second, achieving up to approximately 2.5 terabytes per second of aggregate bidirectional bandwidth with predictable communication timing. This design is particularly suitable for scenarios where communication latency often determines overall performance .

Why Did NVIDIA Acquire Groq's Technology Instead of Building In-House?

Groq has been building LPU chips independently for several years with a focus on deterministic, high-throughput inference. Rather than developing competing technology, NVIDIA licensed the architecture and integrated it into its data center product portfolio as a complementary inference tier. This approach allowed NVIDIA to address a specific market need without diverting engineering resources from GPU development .

The memory wall in GPU inference is real and fundamental. As models grow larger and inference workloads become more diverse, the bottleneck shifts from compute to memory bandwidth. GPUs are general-purpose processors designed to handle training, fine-tuning, vision, video, embedding generation, and inference all on the same hardware. That generality requires programmable compute, high-bandwidth memory for large model storage, and the ability to handle arbitrary input and output shapes. The LPU gives up all of that flexibility to optimize the one operation that accounts for most real-time AI serving cost .

The licensing arrangement also reflects broader industry trends. NVIDIA recognized that specialized architectures designed for specific workloads are becoming key complements to general-purpose GPUs. In AI inference, GPUs are no longer the sole optimal solution; the future of data center infrastructure involves heterogeneous systems where different chips handle different stages of the inference pipeline .

For data center operators, the Groq 3 LPU represents a new option in the inference acceleration toolkit. Whether it makes economic sense depends on your specific workload profile, model sizes, and power constraints. But the underlying architectural principle is clear: as AI workloads mature and scale, task-specific optimization is increasingly valuable compared to general-purpose compute.