policy and the prophetic voice: generative AI and deepfake nudes

This is a mess of a post blending thoughts on tech policy with religious ideas and lacking the kind of obvious throughline or structure that I’d like it to have. It’s also been in my head for a couple of weeks, and it’s time to release it into the world rather than wait for it to be something better. So, here it is:

I am frustrated with generative AI technology for many reasons, but one of the things at the top of that list is the knowledge that today’s kids are growing up in a world where it is possible—even likely—that their middle and high school experiences are going to involve someone using generative AI tools to produce deepfake nudes (or other non-consensual intimate imagery—NCII) of them. See, for example, this horrifying story from the New York Times last April.

Of course, this isn’t limited to kids. As Emanuel Maiberg recently reported at 404 Media:

The most popular tool I’ve seen people use to create nonconsensual intimate videos is Pixverse, which is made by a Beijing-based company called AIsphere and founded by Wang Changhu, the former “head of vision technology” at TikTok owner ByteDance. People who create nonconsensual content use different AI video generators for different purposes, and they commonly use Pixverse to create videos that make female celebrities look as if they’re taking their tops off.

Maiberg’s article is depressing but a must-read—this podcast recap is also a good introduction to the subject. Maiberg’s specific emphasis is that Chinese generative AI video creation tools lack some of the guardrails that we see in American companies, but while that’s a problem in and of itself, I’m more concerned by what we’ve let out of Pandora’s box. If these particular companies introduce more guardrails, aren’t other tools going to pop up? Here’s Maiberg in a previous article, commenting on how open source generative AI models inherently lend themselves to this sort of thing:

On Tuesday, Chinese tech giant Alibaba released a new “open” AI video generation model called Wan 2.1 and shared the software on Github, allowing anyone with the technical know-how and hardware to use and modify freely. It took about 24 hours for the model to be adopted by the AI porn hobbyist community, which has already shared dozens of short AI porn videos using Alibaba’s software. Elsewhere, in a community that’s dedicated to producing and sharing nonconsensual AI-generated intimate media of real people, users are already salivating over how advanced the model is.

This is the double-edged sword of releasing open AI models that users can modify, which on one hand democratizes the use of powerful AI tools, but on the other is often used by early adopters to create nonconsensual content.

So, as long as there’s one open source model out there, there’s going to be the potential to do this kind of thing with video generating AI tools.

One of the most depressing things about reading Maiberg’s first article is that one of the next articles in my “tech news” RSS folder was Adi Robertson’s article The Take It Down Act isn’t a law, it’s a weapon. Here are the first two paragraphs, though I recommend reading the whole thing (or, again, listening to the podcast recap here):

It’s internet safety law season again. After a narrow failure to pass the Kids Online Safety Act in 2024, Congress is now advancing the Take It Down Act, which criminalizes nonconsensual intimate imagery (NCII, once dubbed “revenge porn,” including AI-generated content) and sets requirements for web platforms to remove it. The bill has gained support from First Lady Melania Trump, and President Donald Trump touted it during his joint address to Congress on March 4th, promising he would sign it. In a normal world, this could be a positive step towards solving the real problem of NCII, a problem that AI is making worse.

But we are not in a normal world. Parts of the Take It Down Act are more likely to become a sword for a corrupt presidential administration than a shield to protect NCII victims — and supporters of both civil liberties and Big Tech accountability should recognize it.

My thinking on tech policy has gone through a lot of shifts over the past decade. As I started to see the harms that tech can do, I got more and more keen on regulating technology and technology companies. More recently, though, voices like Mike Masnick at Techdirt and Cory Doctorow and others at the EFF have tempered some of that enthusiasm, calling more attention to how well-intentioned policy can be misused in ways that we ought to be wary of.

Robertson’s article makes it clear how pressing such skepticism is under a second Trump administration, which is pretty clear about its intent to use the law in ways that benefit its priorities while ignoring (or perpetuating) obvious hurts and harms whose resolution would not benefit them. It’s about how much it sucks to live in a world where generative AI makes NCII so damn easy to produce but where giving the government the power to punish the creation of AI-generated NCII might actually be worse. It’s a lot. It sucks a lot.

I’m not a tech policy expert, and I’m not a lawyer. There may well be a good way to craft a policy response to AI generated deepfake nudes that doesn’t further empower Trump, Musk, and company to target political enemies and make bad faith defenses of their allies. The point of this post is not to advocate any kind of policy or legal nihilism. Rather, it’s to argue that we need more than policy—we need prophetic voice.

A couple of years ago, I wrote a post that quoted from the introduction to Thomas Merton: Passion for Peace, and I want to repeat that quote here:

Merton felt that, however poorly equipped he might be for the role, he was called to be a prophet. He understood clearly the limitations of the prophet’s vocation. A prophet is not necessarily one who has all the right answers; he or she is the one who knows at a given moment in history what the real problems are that humanity must face, what the proper goals are that need to be achieved, what the right questions are that must be asked. In a passage from Conjectures of a Guilty Bystander Merton offers a perceptive explanation for the popularity of his writings. “It seems to me that one of the reasons why my writings appeal to many people is precisely that I am not so sure of myself and do not claim to have all the answers” (p. 49). In his writings on war and peace, it is passion for the goals he sees rather than certitude about the way to achieve them that dominates. There were times that he felt lonely, diffident, distrustful of himself, even shocked by some of the things he heard himself saying.

I don’t know what the “right answers” are to deepfake nudes and all of the other societal problems posed by generative AI, but I think we need more people who are willing to speak up for those “real problems” and articulate that “humanity must face” them. What’s more, I think that most “right answers”—including more technological safeguards at the company level or carefully thought-out policies at the government level—are necessarily incomplete. We need prophetic calls and prophetic responses, a willingness at the level of individuals and communities to take radical action in our own lives to stand against these real problems and not perpetuate them.

Here’s Christian anarchist Leo Tolstoy, in The Kingdom of God is Within You, articulating a radical confidence that humanity is capable of setting aside its harmful tools to bring about a better world:

Thus the prediction that the time will come when all men shall be instructed by God, shall stop warring, shall forge the swords into plowshares and the spears into pruning-hooks, that is, translating into our language, when all the prisons, fortresses, barracks, palaces, churches, shall remain empty, and all the gallows, guns, cannon, shall remain unused, is no longer a dream, but a definite, new form of life, toward which humanity is moving with ever increasing rapidity.

[…]

All we can know is, what we, who compose humanity, must do, and what not, in order that the kingdom of God may come. That we all know. And everyone need but begin to do what we must do, and stop doing what we must not do; every one of us need only live by all that light which is within us, in order that the promised kingdom of God, toward which the heart of every man is drawn, may come at once.

I don’t know that I share Tolstoy’s confidence that we can get to a point where our technologies of harm “remain unused,” but I agree with his urgency that we must work in that direction. Likewise, I agree with Jacques Ellul, another Christian anarchist, who writes that the struggle for this better world is essential, even if it may be impossible to ever achieve it.

I’m not very good at being prophetic or radical, but one of the reasons that I’ve stubbornly refused to use generative AI tools is that it gives me a chance to try. I have no doubt that these tools, if responsibly used, could be tremendously helpful for me in my personal and professional lives. I know plenty of people who do use these tools in these ways, and I generally trust them to do so. For me, though, if I don’t pick up this sword, I’ll never have to beat it into a plowshare. I am inescapably intertwined with so many technological systems that are making the world a crummier place, and if I can avoid participating in the technological ecosystem that will inevitably make deepfake nudes (and so many other problems) a part of kids’ and adults’ lives today, I’m going to do it.

• NCII
• generative AI
• tech policy
• prophetic people
• prophetic voice
• 404 Media
• Emanuel Maiberg
• Adi Robertson
• The Verge
• Techdirt
• Mike Masnick
• Cory Doctorow
• EFF
• Donald Trump
• Elon Musk
• Thomas Merton
• Thomas Merton: Passion for Peace
• Leo Tolstoy
• The Kingdom of God is Within You
• Christian anarchism
• Jacques Ellul
• Anarchie et christianisme
• kingdom of God