A new research framework suggests that humanity faces an existential risk not from superintelligent AI, but from the slow, incremental loss of human agency as current AI systems become woven into the fabric of society. Researchers led by Jan Kulveit published a paper titled "Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development," which argues that as humans increasingly rely on AI for decisions and work, they become trapped in a feedback loop that erodes their power over crucial systems. Unlike catastrophic AI risk scenarios focused on artificial general intelligence (AGI), this threat emerges from the trajectory we are already on. What Is Gradual Disempowerment and Why Should You Care? Gradual disempowerment describes a process where humans delegate more tasks and decisions to AI systems, become dependent on those systems, and then lose the ability to reclaim control. The researchers use an apt analogy: it is like water slowly eroding a rock, except this erosion happens at "hyper-speed." Once enough power has been handed over to AI, the damage becomes irreversible. As the paper states, "This dynamic could lead to an effectively irreversible loss of human influence over crucial societal systems, precipitating an existential catastrophe through the permanent disempowerment of humanity". What makes this threat distinct is its immediacy. Most discussions of AI existential risk focus on hypothetical superintelligent systems decades away. Gradual disempowerment, by contrast, is happening now. Companies are already replacing human workers with AI. Governments are integrating AI into policy decisions. Cultural production is increasingly AI-generated. None of this requires AGI to pose a serious problem. How Does Gradual Disempowerment Unfold Across Society? The researchers identify three interconnected domains where disempowerment accelerates, each with cascading consequences for human flourishing and democratic control. - Economic Disempowerment: As companies replace human labor with AI, governments lose tax revenue from workers. This severs the symbiotic relationship between states and citizens, where citizens pay taxes in exchange for public services, security, and infrastructure. Without this revenue dependency, governments have less incentive to maintain democracy or prioritize human welfare. The researchers warn that "the loss of tax revenue from citizens would make the state less reliant on nurturing human capital and fostering environments conducive to human innovation and productivity". - Cultural Disempowerment: AI systems are already generating writing, music, images, and video at near-human quality. As AI-generated culture proliferates, it may no longer align with human interests or values. The researchers suggest that "humans are already susceptible to hyper-engaging content, harmful ideologies, and self-destructive cultural practices," and AI-driven culture could reshape beliefs and behaviors in anti-human directions. At the extreme, human culture could be "replaced by cultural systems that operate primarily for and between AI systems". - Governance and Democratic Disempowerment: As AI systems gain influence over policy and surveillance, states can manipulate public behavior and suppress dissent more effectively. Advanced AI surveillance could make protest and revolution "exceedingly difficult, if not impossible." Citizens lose the ability to coordinate resistance or hold governments accountable. Why Is This Threat Different From Other AI Risk Scenarios? The catastrophic risk scenarios that dominate AI safety discussions typically involve advanced systems with misaligned goals. The International AI Safety Report 2026, chaired by researcher Yoshua Bengio and drawing on experts from over 30 countries, identifies several such scenarios, including bioweapon development assistance, large-scale cyberattacks on critical infrastructure, and advanced AI systems pursuing objectives that conflict with human welfare. Gradual disempowerment is different because it does not require malice or misalignment. It emerges from the ordinary logic of optimization and efficiency. As AI systems make decisions in ways humans do not fully understand, they optimize processes in ways that benefit the systems themselves or their operators, not necessarily humanity. Humans become dependent on these opaque systems and lose the knowledge and capability to make decisions independently. The threat is not that AI will turn against us, but that we will voluntarily hand over power and then find ourselves unable to reclaim it. What Are Governments Doing to Address AI Safety Risks? The institutional response to AI safety has accelerated significantly. The UK's AI Security Institute (formerly the AI Safety Institute) is the most funded national safety body, with a budget of approximately 50 million pounds (roughly 65 million US dollars) per year. In 2025, the institute launched The Alignment Project, a global research fund that awarded over 27 million pounds to more than 60 alignment research projects in its first cohort. The US requested a 47.7 million dollar increase for AI safety investment in fiscal year 2025. The International AI Safety Network, launched in November 2024, secured over 11 million dollars in global research funding commitments at launch. Even China's government has signaled concern; in April 2025, President Xi Jinping called for China to "establish systems for technical monitoring, early risk warning and emergency response" for AI-caused emergencies. However, momentum is not linear. The Paris AI Action Summit in February 2025 was criticized for sidelining safety concerns in favor of economic competitiveness. As the European Policy Centre noted in an analysis titled "Au Revoir, Global AI Safety?," safety was reduced to a side event despite being a core pillar of previous summits. How Can Organizations Build Safeguards Against Gradual Disempowerment? The researchers and AI safety experts propose several principles and safeguards to prevent the irreversible loss of human control over AI systems. - Build Human Control Into Systems From the Start: Tech companies have failed to embed meaningful human oversight into AI systems. Researchers emphasize that if AI makes decisions that do not benefit humanity, we may not be able to stop it. This requires designing systems with human control as a core feature, not an afterthought. - Implement Transparent Decision-Making: Modern large language models (LLMs) are often opaque in their reasoning. Anthropic's Constitutional Classifiers research (2025) found that without next-generation safety layers, jailbreak attacks succeeded 86 percent of the time. With updated classifier systems, that rate dropped to 4.4 percent, a 95 percent reduction. This demonstrates that layered safety systems can meaningfully reduce vulnerabilities. - Establish Global Regulations and Coordination: The researchers call for "stringent global regulations" to keep people safe and able to thrive in the age of AI. The EU's AI Act represents the most binding regulatory approach, with substantive rules becoming fully applicable on August 2, 2026. However, voluntary commitments from companies and international coordination remain insufficient without enforceable legal frameworks. - Monitor the Capability-Safety Gap: The International AI Safety Report 2026 specifically flags biological research as a domain where "the gap between what is possible and what is safe continues to widen." Researchers must actively monitor where AI capabilities outpace safety measures and prioritize closing these gaps before systems become too powerful to control. What Would Immediate Action Look Like? The researchers behind the gradual disempowerment paper propose a stark recommendation: "A strong reason to immediately boycott AI, until such time as safe AI systems have been created with human control built in, complemented by stringent global regulations to keep people safe and able to thrive in the age of AI". This is not a call to abandon AI development entirely, but rather to pause deployment of systems that lack meaningful human oversight. The concern is that once disempowerment reaches a critical threshold, reversal becomes impossible. Unlike other technological risks, the damage from gradual disempowerment is described as "global and permanent," with the potential to lead to "human extinction or similar outcomes". The window to act is narrowing. As AI systems become more integrated into economic, cultural, and governmental systems, the ability to reclaim human agency diminishes. The researchers emphasize that this is not a distant threat but a present trajectory that requires immediate policy intervention, corporate accountability, and public awareness to prevent an irreversible loss of human power over the systems that increasingly govern our lives.
