The biggest constraint facing AI infrastructure today isn't computing power or chip availability, it's access to electricity. As enterprises grapple with skyrocketing energy demands from artificial intelligence, a quiet shift is underway: companies are moving AI workloads away from centralized mega-data centers and closer to where their data actually lives. This decentralization strategy could reshape how organizations balance innovation with sustainability .

Why Is Power Becoming the Bottleneck for AI Infrastructure?

The numbers tell a sobering story. The International Energy Agency (IEA) predicts that energy demand from data centers will more than double by 2030, with electricity demand from AI-optimized data centers projected to more than quadruple by 2030 . Graphics processing units (GPUs), the chips that power AI model training and inference, are becoming increasingly power-hungry. Nvidia's roadmap assumes the 1 megawatt (MW) rack is not far away, while the company is championing a transition from 48V or 54V direct current (DC) at the rack to 800V DC power for data centers .

For infrastructure operators, this creates a real problem. Nscale, a European cloud operator, has identified power access as its single biggest constraint. "That is the biggest constraint that we see," noted Tom Burke, Nscale's chief revenue officer, during a recent industry event . The company's data center network is centered on Norway, where cold climate and abundant hydroelectric power offer distinct advantages when running power-hungry, heat-generating AI infrastructure .

The power demands of GPUs have triggered broader infrastructure changes. Burke explained that the heat transfer requirements of chips drove a transition from air-cooled data centers to liquid-cooled data centers, and with that came accelerated innovation cycles from GPU manufacturers . What used to be two-year release cycles have consolidated down to six-month release cycles because of how fast innovation is moving in response to cooling and power challenges .

How Are Enterprises Reducing Energy Consumption Through Distributed AI?

Rather than relying solely on hyperscaler cloud providers, enterprises are exploring a different path: running AI workloads locally or at the edge. This approach addresses multiple challenges simultaneously. Rabih Bashroush, professor of digital infrastructure at the University of East London, notes that there is significant open-source AI that companies are downloading and running internally . Many enterprises are also shifting toward specialized models, which are much more efficient than general-purpose models offered by large AI companies .

The efficiency gains from this approach are substantial. Organizations adopting modern, software-defined infrastructure have reported energy reductions of around 50% compared to legacy environments, according to James Sturrock, director of systems engineering at Nutanix . Even simpler strategies exist for efficiency when running smaller models using less data away from hyperscalers' infrastructure. Bashroush noted that once you run AI this way, "you're switching it off when there's no one in the office" .

Bashroush

Edge locations offer another advantage: they typically have pre-existing power infrastructure. As Karim Abou Zahab, principal for sustainable transformation at Hewlett Packard Enterprise (HPE), explained, the data center boom means getting a new grid connection requires waiting in line for years . Enterprises are increasingly looking at where AI runs and how efficiently it can be deployed closer to their data and operations .

Steps to Optimize AI Energy Consumption in Your Organization

• Assess Workload Location: Evaluate whether your AI workloads truly need centralized cloud infrastructure or if they can run efficiently at edge locations with existing power infrastructure and lower latency requirements.
• Prioritize Software Efficiency: Implement software-driven optimization to ensure compute is fully utilized and energy isn't wasted through idle or over-provisioned infrastructure, which can reduce energy consumption by approximately 50%.
• Consider Specialized Models: Move away from general-purpose AI models toward specialized, smaller models tailored to your specific use cases, which consume significantly less power and can be run locally or at the edge.
• Evaluate Infrastructure Modernization: Upgrade to modern, software-defined infrastructure that supports efficient resource allocation and can be powered down when not in use, rather than maintaining constant operation.

What Role Does Hardware Optimization Play in Reducing Data Center Power?

Hardware providers are shifting their product focus to meet demand for decentralization. Supermicro, a major server manufacturer, has introduced compact, energy-efficient systems designed to accelerate adoption of intelligent edge AI . These purpose-built edge AI systems support real-time inferencing and business-critical workloads across retail, manufacturing, healthcare, and enterprise environments, with flexible configurations that deliver data center-class performance in space- and power-constrained edge deployments .

Smart networking infrastructure also plays a critical role. Vast Data highlighted the impact of Nvidia's Bluefield 4 Smart Network Interface Cards (NICs), which can carry storage software platforms. According to Vast cofounder Jeff Denworth, "For every 1,100 GPUs, you don't have to deploy another 256 physical Vast C node servers. So, your cost saving is off the charts. Your power saving is also quite considerable. We can reduce power for your infrastructure by about 75%" .

Vast Data

"For every 1,100 GPUs, you don't have to deploy another 256 physical Vast C node servers. Your power saving is also quite considerable. We can reduce power for your infrastructure by about 75%." said Jeff Denworth, cofounder at Vast Data.

Jeff Denworth, Cofounder at Vast Data

The broader implication is that IT decision-makers must treat efficiency as a factor from the outset. Zahab emphasized that this means looking at the entire IT estate: "The data fed into models, the software used to interact with and train them, the right equipment, data center resources, and the energy sources powering them" .

Zahab

Is This Shift Enough to Address the AI Energy Crisis?

While distributed AI infrastructure offers real efficiency gains, experts caution against assuming the problem is solved. The IEA estimated that in 2024, AI was still only responsible for 15% of data center energy demand, with most demand coming from standard compute workloads . However, inferencing energy use, the process of running trained AI models on new data, is set to outpace training. Inferencing is projected to almost double by 2030, with consumption reaching 162.5 terawatt-hours (TWh) .

This creates both a challenge and an opportunity. Zahab noted that this gives enterprises a runway to reduce costs and carbon footprints if efficiency is prioritized from design through to deployment . The key insight is that enterprise AI workloads represent only a fraction of total cloud and data center workloads, meaning there's significant room for optimization before hitting hard physical limits .

The shift toward distributed, edge-based AI infrastructure reflects a fundamental recognition: the centralized mega-data center model, while powerful, may not be the most sustainable or practical path forward for all organizations. By moving AI closer to where data originates and combining that with specialized models and efficient hardware, enterprises can achieve meaningful energy reductions while maintaining the computational power needed for modern AI applications.