The energy crisis facing AI data centers isn't about training massive models like GPT-4, but rather the billions of everyday requests users make to AI systems every single day. Over 80% of AI computing is now used for inference, the process of running trained models to generate responses, according to recent research compiled by energy efficiency experts. This shift fundamentally changes how companies should approach reducing AI's environmental footprint, and it's already attracting serious attention from major tech firms and university researchers. Why Is Inference Consuming So Much More Energy Than Training? When you ask an AI chatbot a question or generate an image, you're triggering inference. When researchers spend weeks training a new model from scratch, that's training. While training gets the headlines, inference happens constantly across millions of devices and applications. A single large language model might be trained once, but it answers thousands of queries every second across the globe. The numbers are striking. Text responses from smaller models like Llama 3.1 8B consume roughly 114 joules per response when accounting for all the computing overhead involved. Larger models like Llama 3.1 405B use about 6,706 joules per response. Video generation is far more demanding, with higher-quality 5-second videos requiring approximately 3.4 million joules. These individual tasks might seem small, but multiplied across billions of daily requests, they create enormous aggregate energy demands. How Can Companies Actually Reduce AI's Energy Footprint? Energy researchers have identified concrete strategies that companies can implement immediately. Rather than chasing marginal improvements in how models are trained, the focus should shift to making inference more efficient across the board. - Prioritize Inference Efficiency: Treat energy per inference as the primary optimization target, focusing on high-frequency endpoints that handle the most requests rather than rare use cases that few people access. - Use Specialized Models for Narrow Tasks: Deploy task-specific models for classification, ranking, and extraction instead of using large generative models for everything, since specialized models consume significantly less energy and carbon. - Measure Energy Per Task: Instrument data center pipelines to measure actual energy consumption for specific tasks like text, image, and video generation, including all non-GPU overhead from memory, networking, and orchestration. - Optimize Hardware Utilization: Increase accelerator utilization through batching requests together, caching results, implementing smarter scheduling, and eliminating redundant calls across different systems and services. - Design for Hardware Efficiency: Create models that fit efficiently within memory and bandwidth constraints of existing hardware, and maximize utilization of current equipment before scaling up capacity. The transparency gap remains a significant challenge. Most major AI model providers do not disclose sufficient information to estimate their total energy use or carbon footprint reliably, making it difficult for customers to understand the true environmental cost of their AI usage. What Are Universities and Tech Companies Doing Right Now? Carnegie Mellon University researchers are developing hardware solutions that could dramatically reduce data center energy demands. Assistant Professor Akshitha Sriraman and her team are designing what they call "carbon-efficient servers" that blend new and old technology by reusing older server components while incorporating much more energy-efficient new parts. "We are computer architects and systems researchers, which means that we try to figure out how we can design the data center hardware devices in ways that are more efficient and sustainable," explained Akshitha Sriraman, Assistant Professor of Electrical and Computer Engineering at Carnegie Mellon University. Akshitha Sriraman, Assistant Professor of Electrical and Computer Engineering, Carnegie Mellon University The impact could be substantial. According to Sriraman, widespread adoption of these efficient servers by large cloud companies could eliminate roughly 100 million of the 2.5 billion metric tons of carbon emissions that the cloud is projected to emit by 2030, equivalent to eliminating the annual emissions from entire countries like Qatar or Venezuela. Microsoft is already exploring adoption of these designs for both internal operations and public cloud customers as part of its 2030 decarbonization targets. Another Carnegie Mellon initiative takes a different approach. Professors Brandon Lucia and Nathan Beckmann created a brand new type of chip architecture through their company, Efficient Computer, which recently announced $60 million in new funding. Their processor eliminates the constant need to fetch new instructions from memory and improves how data flows within the chip, reducing energy consumption dramatically. "We are 10 times more energy efficient than the best low-power general purpose computers on the market today," stated Brandon Lucia, CEO of Efficient Computer. "Meaning, if you hook ours up to a battery, hook theirs up to a battery, you just run a general purpose computation over and over, ours will last 10 times longer. A few weeks become years." Brandon Lucia, CEO of Efficient Computer and Electrical and Computer Engineering Professor, Carnegie Mellon University A third CMU researcher, Peter Zhang, is exploring whether data centers could shift their workloads to operate during off-peak hours when energy demand is lower. His proposal for "nocturnal data centers" won the inaugural AI and Energy seed grant, suggesting that dynamic workload adjustments could help stabilize electricity demand profiles and reduce strain on the country's aging energy grid. His What Do These Findings Mean for Energy Grids and Consumers? The scale of AI's energy demands is growing rapidly. Projections suggest AI will use over half of all data center electricity by 2028, and AI-specific servers are estimated to have consumed between 53 and 76 terawatt-hours in 2024, with projections reaching 165 to 326 terawatt-hours by 2028. For context, data centers overall are responsible for just over 1% of global electricity demand today, but that share is expected to grow significantly. The financial impact is already being felt. Rising energy demands from data centers are driving up utility bills for Americans across the country, according to the U.S. Energy Information Administration. The U.S. Department of Energy projects that AI energy demands could represent as much as 12% of the country's total energy consumption by 2028, with electricity demands potentially doubling or tripling in the next few years. The good news is that efficiency improvements are already happening. Google reported that over the past 12 months, the energy per median Gemini prompt fell by 33 times, and the total carbon footprint fell by 44 times. Each text prompt to Gemini now uses just 0.24 watt-hours of energy and produces 0.03 grams of carbon dioxide equivalent, with water consumption at 0.26 milliliters per prompt. The path forward requires a fundamental shift in how the industry approaches AI efficiency. Rather than focusing solely on making training more efficient, companies need to optimize the billions of daily inference tasks that power AI features in products and services. With hardware innovations, better measurement practices, and strategic workload management, researchers believe the AI industry can dramatically reduce its environmental footprint while continuing to deliver the AI capabilities that users expect.
