Instead of building more energy-hungry data centers, a growing movement in AI is training models across geographically dispersed networks of existing computers, from research labs to solar-powered homes. This decentralized approach harnesses idle computing capacity and renewable energy sources rather than concentrating all training in one location, offering a path to make artificial intelligence significantly more sustainable .

Why Is AI Training So Energy-Intensive?

Training large language models (LLMs), which are AI systems that understand and generate human language, represents one of the most power-hungry phases in a model's lifecycle. As these models grow larger and more capable, even massive single data centers struggle to keep up with the computational demands. Tech companies are increasingly turning to the pooled power of multiple data centers across different geographic locations to handle the scale required for modern AI training .

The problem is accelerating. Nvidia, the dominant maker of graphics processing units (GPUs) used for AI training, launched the Spectrum-XGS Ethernet for scale-across networking, which "can deliver the performance needed for large-scale single job AI training and inference across geographically separated data centers." Similarly, Cisco introduced its 8223 router designed to "connect geographically dispersed AI clusters," signaling that the industry recognizes centralized training is reaching its limits .

How Can Decentralized Training Reduce Energy Consumption?

Decentralization works by allocating model training across a network of independent nodes rather than relying on one platform or provider. The key insight is elegantly simple: instead of moving energy to where AI is being trained, companies can move AI training to where energy already exists. This approach harnesses computing power from dormant servers in research labs, underutilized GPUs in corporate offices, and even consumer devices in solar-powered homes, avoiding the need to construct new data centers that require electric grids to expand their infrastructure .

A peer-to-peer cloud computing marketplace called Akash Network has emerged as a practical example of this model. Billed as the "Airbnb for data centers," Akash allows organizations with unused or underused GPUs to register as providers, while those needing computing power can rent access. This creates a marketplace for idle compute capacity that would otherwise sit unused .

"If you look at AI training today, it's very dependent on the latest and greatest GPUs. The world is transitioning, fortunately, from only relying on large, high-density GPUs to now considering smaller GPUs," said Greg Osuri, cofounder and CEO of Akash Network.

Greg Osuri, Cofounder and CEO at Akash Network

What Technical Innovations Make Distributed Training Possible?

Decentralized AI training requires both hardware and software innovations. On the software side, federated learning, a form of distributed machine learning, enables organizations to collaborate on training without sharing raw data. In this approach, a central server distributes an initial model to participating organizations, which train it locally on their own data and share only the model weights (the numerical parameters that define the model's behavior) back to the central server. The server aggregates these weights, typically by averaging them, and sends the updated model back to participants. This cycle repeats until the model is fully trained .

However, this constant back-and-forth communication creates challenges. High communication costs and fault tolerance issues arise when nodes fail during training. To address these problems, researchers at Google DeepMind developed DiLoCo, a distributed low-communication optimization algorithm that forms what the researchers call "islands of compute." Each island consists of a group of chips of the same type, and islands are decoupled from each other, allowing them to perform training steps independently without constant communication. If a chip fails, only its island is affected, not the entire training process .

An improved version called Streaming DiLoCo further reduces bandwidth requirements by synchronizing knowledge "in a streaming fashion across several steps and without stopping for communicating," similar to watching a video that hasn't fully downloaded yet. This innovation has already been adopted by real-world projects. Prime Intellect implemented DiLoCo to train a 10-billion-parameter model across five countries spanning three continents, while 0G Labs adapted the algorithm to train a 107-billion-parameter foundation model across a network of segregated clusters with limited bandwidth .

Steps to Implement Decentralized AI Training in Your Organization

• Assess Idle Capacity: Inventory existing computing resources across your organization, including underutilized GPUs in offices, research labs, and data centers that could participate in distributed training networks.
• Evaluate Federated Learning Frameworks: Explore distributed machine learning platforms and frameworks like PyTorch, which now includes DiLoCo in its repository of fault tolerance techniques, to understand how to structure collaborative training.
• Plan for Redundancy and Monitoring: Design systems with built-in fault tolerance so that individual node failures don't interrupt the entire training process, and implement monitoring to track performance across geographically dispersed clusters.
• Consider Energy Sources: Identify opportunities to connect training infrastructure to renewable energy sources or locations with lower-cost electricity, maximizing the sustainability and cost benefits of decentralization.

What Does the Future of Decentralized AI Look Like?

Akash Network is pursuing an ambitious vision through its Starcluster program, which aims to convert solar-powered homes into functional data centers for AI training. The program targets reaching this goal by 2027 and plans to expand beyond homes to schools and community sites. Participants would need solar panels, consumer-grade GPUs, batteries for backup power, and redundant internet connections to prevent downtime. Akash is collaborating with industry partners to subsidize battery costs and make participation more accessible .

"We want to convert your home into a fully functional data center," said Greg Osuri, emphasizing the vision of distributing AI training infrastructure to where renewable energy is already available.

Greg Osuri, Cofounder and CEO at Akash Network

Researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) see significant promise in this approach. Lalana Kagal, a principal research scientist at MIT CSAIL who leads the Decentralized Information Group, noted that decentralized training offers the option of training models "in a cheaper, more resource-efficient, more energy-efficient way." While these training methods are more complex than traditional centralized approaches, they provide a compelling tradeoff: companies can use data centers across distant locations without needing to build ultrafast, expensive bandwidth connections between them, and fault tolerance is built in because the impact of a chip failure is limited to its island of compute .

The broader implication is significant. Rather than continuously constructing new energy-hungry data centers to meet AI's growing computational demands, organizations can leverage existing underutilized processing capacity worldwide. As Osuri frames it, the future of sustainable AI lies in moving "AI to where the energy is instead of moving the energy to where AI is" .