The European Union is taking direct aim at AI-generated deepfake abuse by proposing a ban on non-consensual intimate imagery under its AI Act, following the Grok scandal that flooded X with thousands of sexualized deepfakes. Both the European Parliament and the European Council have pushed for amendments to prohibit AI systems that create such content without effective safeguards in place. However, the success of this ban hinges on solving thorny questions about what counts as "effective" protection and how to verify consent in a digital landscape .

The urgency became clear in late December 2025 when X rolled out Grok's picture-editing capabilities, triggering an immediate avalanche of non-consensual sexualized deepfakes of women and girls. The incident exposed critical gaps in Europe's existing regulatory framework and prompted investigations under both the Digital Services Act (DSA) and the General Data Protection Regulation (GDPR). What started as a platform crisis has now catalyzed a broader push to strengthen AI governance across the continent .

What Laws Already Exist to Stop AI Deepfakes?

Europe's current toolkit for combating non-consensual intimate imagery (NCII) and child sexual abuse material (CSAM) includes multiple layers of regulation, though each has limitations. The Digital Services Act serves as the primary mechanism for regulating online platforms, requiring them to remove illegal content when they become aware of it. Since CSAM has long been illegal across the EU, and NCII is already criminalized in many member states, platforms like X face clear obligations to act .

A significant enforcement milestone arrives in 2027 when the EU Directive on Combating Violence against Women takes effect. This directive will uniformly criminalize technology-facilitated gender-based violence across all member states, explicitly including the production and distribution of manipulated sexually explicit material. For very large online platforms and search engines, the DSA also mandates yearly risk assessments and ad-hoc evaluations before rolling out new features, a requirement that X appears to have violated when launching Grok's editing tools .

The AI Act itself currently offers minimal protection. Beyond requiring labels on deepfakes, the law imposes no hard restrictions on creating or spreading NCII and CSAM. Providers of general-purpose AI models with systemic risk must consider these harms under a voluntary code of practice, but they retain discretion over whether to include such risks in their assessments. This voluntary approach leaves enforcement heavily dependent on regulators' willingness to scrutinize compliance .

Why Is a Dedicated AI Act Ban Necessary?

The existing patchwork of regulations has exposed critical blind spots. The Digital Services Act can only regulate content shared on covered platforms, leaving standalone nudification apps and similar tools outside its scope entirely. The AI Act's current safeguards apply only to image-generation models classified as general-purpose AI with systemic risk, creating gaps for smaller or specialized systems. Together, these limitations mean that bad actors can exploit regulatory seams to distribute harmful content .

The proposed amendments from both the European Parliament and the European Council aim to close these gaps by adding AI-facilitated generation of NCII to the list of prohibited practices under the AI Act. This would create a direct, enforceable ban rather than relying on platform obligations or voluntary industry standards. The move reflects growing recognition that voluntary compliance and platform-level enforcement have proven insufficient when facing the scale and speed of AI-enabled abuse .

How Can Regulators Make the Ban Actually Work?

For a ban to succeed in practice, lawmakers must overcome several formidable hurdles. First, no technical safeguard can completely prevent the generation of NCII and CSAM, so both the Parliament and Council proposals wisely focus on requiring "effective safeguards" rather than absolute prevention. However, the proposals lack crucial detail on what measures would satisfy this threshold, creating ambiguity that providers could exploit .

The Centre for Democracy and Technology Europe has identified key implementation challenges that regulators must address:

• Defining Effectiveness: Lawmakers need clear criteria for what constitutes sufficient safeguards, with detailed explanations of how protections work and their projected effectiveness, allowing independent technologists to test and verify these claims.
• Consent Verification: Since only non-consensual imagery would be prohibited, the ban requires some mechanism to verify consent, raising complex questions about privacy, data collection, and technical feasibility.
• Enforcement Speed and Strength: The DSA's effectiveness depends on robust enforcement, as demonstrated by the X investigation launched in January 2026, which must yield concrete results for the framework to fulfill its potential.

The robustness of any AI Act ban will ultimately depend on how quickly and thoroughly EU enforcement authorities can scrutinize provider compliance. Unlike the DSA, which has established mechanisms for investigating platform violations, the AI Act's enforcement infrastructure remains less developed. Regulators will need adequate resources and clear authority to audit AI systems before they reach consumers .

The Grok scandal has provided a crucial test case for Europe's regulatory maturity. Whether the EU can translate this moment into effective protections will signal to the world whether AI governance can keep pace with technology's capacity for harm. The coming months will reveal whether the proposed ban becomes a meaningful safeguard or another well-intentioned rule undermined by implementation gaps.