San Francisco city attorney David Chiu has sent cease-and-desist letters to Apple and Google, demanding they remove 13 face-swapping apps that can create nonconsensual nude images. The letters claim the companies have likely made millions of dollars in fees from these apps and should improve their moderation processes to prevent their presence in app stores. Chiu emphasized that generating non-consensual intimate images is illegal, harmful, and completely unacceptable. The city attorney’s office previously took legal action against 16 popular deepfake websites, and the current legal notices assert that California’s laws prohibit supporting services that create deepfake pornography. The apps use in-app payments, which the tech companies take a cut of, the letters say. 'The fact that some of the world’s largest and most established technology companies are facilitating this has to stop,' Chiu said.
Researchers have repeatedly found and reported apps in Apple’s App Store and Google’s Play Store that allow people to generate sexual images using AI—including some apps being rated as suitable for use by children. While new laws and bans aim to tackle the scourge of explicit deepfakes online, technology and social media companies consistently direct millions of people toward the harmful tech. Both Apple and Google have developer policies that prohibit pornography, abuse, and harassment on their platforms. They have previously removed dozens of nudify and deepfake apps, after reports by researchers and journalists. Google spokesperson Dan Jackson told WIRED that the company has deleted 'hundreds' of apps with nudifying features for policy violations, including the five Android apps flagged by Chiu’s office, among other steps to restrict access to them.
Over the last five years, a highly lucrative slurry of deepfake 'nudification' tech has emerged online—most transparently with xAI’s Grok being used to create millions of sexualized images in January. A host of apps, websites, and bots allow people (mostly men) to upload pictures of people (overwhelmingly women and girls) and digitally 'remove' clothing or place them into graphic sexual scenarios. Often all it takes to create sexual deepfakes is a reference photo and a couple of clicks, with some results available in seconds. Images and videos have become more realistic as the underlying generative AI technology has improved, with services providing some results for free or charging small fees to create the harmful content. Previous reporting by WIRED and Indicator Media has uncovered incidents in at least 90 schools where deepfake sexual abuse images have been created of minors.
Source: wired
