Why is Hugging Face hosting tools to make deepfake porn of teenage celebrities?
The “ethical” AI company is still hosting models designed to make nonconsensual sexual content, despite clear breaches of its policies
Update, 24 July: After we sent Hugging Face the URLs of the models referenced in this article, the company removed some (but not all) of them. See our full update below.
Hugging Face is one of the world’s leading AI companies. Valued at over $4.5bn and backed by investors including Google, Amazon and Nvidia, the platform — which primarily serves as a repository for AI models and datasets — likes to portray itself as the ethical AI company.
In 2022, Hugging Face published a paper on “putting ethical principles at the core of the research lifecycle,” and its chief ethics scientist has described the company as founded on “key ethical values in tech: collaboration, responsibility, and transparency.” Last year a group of Hugging Face executives wrote a piece for Teen Vogue which decried the scourge of nonconsensual deepfake pornography, calling it a “massively harmful practice” that “we must … end … for good.”
But Hugging Face is not living up to those values, particularly when it comes to deepfake porn. 404 Media reported last week that Hugging Face hosts thousands of nonconsensual AI models of real people, including celebrities, many clearly intended to be used to create sexual content. More than a week later they are still up, and Transformer has found at least one instance where a model is explicitly aimed at creating sexual content featuring a teenage-looking likeness of an actress.
Preview images intended to demonstrate the model’s outputs are indistinguishable from real images, and when run through a reverse image search returned numerous results of the actress as a child.
The model has been available on Hugging Face for almost five months, and is still up as of the time of writing. Transformer has chosen not to name the celebrity in question.
Transformer also found that Hugging Face hosts a model designed to generate videos of the celebrity. The model description says “I like her pink dress look,” with reference to a film that was released when the actress was 15. The video model’s file name does not explicitly say it is for creating pornography, but it is stored in a directory that features many other models clearly intended for pornographic use.
Elsewhere on Hugging Face are dozens of other models explicitly designed to generate pornographic images of female celebrities. These have explicit filenames, and are stored in a Hugging Face folder that also contains models designed to generate photos of the celebrities in different sexual positions.
All of this material violates Hugging Face’s content policy, which forbids sexual content “created without explicit consent.” It also forbids “underage nudity or any sexual content involving minors.” In both the US and UK, sharing non-consensual sexual imagery is illegal, and creating it will soon be illegal in the UK. But neither country has explicitly banned distributing tools to produce it.
Finding this content was not difficult. The bulk of the material appears to have been uploaded to Hugging Face as part of an effort to archive CivitAI, the notorious AI platform which recently banned any deepfake-related content or models. A publicly available tool, first reported on by 404 Media, allows anyone to easily search the CivitAI archive and find the models on Hugging Face.
INFO: How the deepfake porn models work
Many of the “models” we talk about here are technically not, in fact, AI models. They are “LoRAs”, or “Low-Rank Adaptations” — a way to fine-tune an image or video generation model to get it to produce images or videos of a particular celebrity. The majority of the LoRAs we discuss in this article are, per their file names or descriptions, explicitly designed to be used with pornographic image generators called “BigLust” and “ReallyBigLust.”
Just a few Hugging Face users appear to have uploaded much of CivitAI’s celeb deepfake porn models, with one prolific user even noting their ties to the CivitAI archive project in their Hugging Face bio. As of the time of publication, Hugging Face has not banned this person’s account, nor those of the other prolific uploaders whose accounts we reviewed. Shortly before publication, we sent a list of all the URLs reviewed for this article to Hugging Face. We will update this article if Hugging Face remove them.
When asked for comment Hugging Face’s head of communications Brigitte Tousignant said: “Our team is always working on identifying content that goes against our guidelines, including content that is directly reported to us. If you know of any such content, we encourage you to either report it directly on the platform or send an email with specifics to: safety@huggingface.co.”
Tousignant did not respond to further questions about whether the company is taking proactive action to address the issue, or simply relying on others to find and report the content — or why Hugging Face hasn’t taken action since 404 Media’s report last week.
In their Teen Vogue piece, Hugging Face executives wrote that when it comes to deepfake pornography, “responsibility lies with society as a whole.” If only the company took responsibility itself.
Update, 24 July: After we sent Hugging Face the URLs of all the models referenced in this article, the company removed some (but not all) of them. Several of the models detailed in this piece are still accessible, including the video model that references the actress when she was 15. Hugging Face has also not banned the prolific users we mentioned, and a brief further investigation by Transformer has found that Hugging Face continues to host many other models from those users which are clearly intended to generate deepfake pornography of celebrities and influencers.
We will send these additional URLs to Hugging Face, and perhaps the company will remove them. But a “responsible” company worth $4.5bn shouldn’t be relying on journalists to keep its platform free of tools that clearly violate its own policies, and cause harm to the women they target.