Nieman Foundation at Harvard
Before my career pivoted to writing the words in news stories, I used to draw the illustrations that ran alongside them. As someone who comes from a background in visuals, I’ve been fascinated by the rise of generative AI text-to-image tools like Stable Diffusion, DALL-E, and Midjourney.
When I learned about how these tools were trained by ingesting literally billions of images from the web, I was surprised to see that some of my own images were part of the training set, included without any compensation to or approval by yours truly.
I’m far from alone. By and large, many artists are not happy about how their work (and their signature styles) have been turned into prompts that deprive them of control and compensation of their artwork. But now, a team of computer science researchers at the University of Chicago wants to level the playing field and arm artists with the tools they need to fight back against unauthorized use of their work in training new AI models.
Their paper describes a new tool called Nightshade which can be used against these powerful image generators. Named after the deadly herb, Nightshade allows anyone to invisibly alter the pixels of an image to “poison” the image. Along with mislabeled metadata, the “poisoning attack” can help generate incorrect results in image generators — such as making the prompt “photo of a dog” generate a photo of a cat.
I spoke with Shawn Shan, a graduate researcher and lead student author of the paper, titled “Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models.” The paper was covered extensively in the media when it dropped on Arxiv.org late last month. But I wanted to learn more about what Nightshade means for the fight over artists’ rights online, and the potential that the tool could touch off an arms race between creators and the developers of AI image generators, whose voracious appetite for data doesn’t figure to be sated any time soon.
The interview was edited for clarity and brevity.
A big company just takes your data and there’s nothing artists can really do. Okay. So, how can we help? If you take my data, that’s fine. I can’t stop that, but I’ll inject a certain type of malicious or crafted data, so you will poison or it will damage your model if you take my data. And we designed it in such a way that it is very hard to separate what is bad data, what is good data from artists’ websites. So this can really give some incentives to both companies and artists just to work together on this thing, right? Rather than just a company taking everything from artists because they can.
But that’s fairly easy to remove, right? It’s very clear to a human, but also to many machine systems that this is not correct. So we did some work where we tried to make a cat image that looks like both a cat to a human, but to the model, it will think this is actually a dog.
My take on this space is that it’s kind of about the power asymmetry. Right now, artists really have very limited power and anything will just help tremendously, right? There may be some collateral damage or some side effects of a certain company doing things, but what we think is this is worth it, just to give artists a tool to fight back.
Another take on this is that some of those entertainment companies, perhaps not Disney, but a small or medium-size game company are also very concerned about AI taking their work. So these can probably also help in those cases as well.
So you know, it’s possible for them to filter them out, say, okay, these are malicious data, let’s not train on them. In some sense, we also win in those cases because they remove the data that we don’t want them to train on, right?
So that’s also kind of a benefit of that case. But I feel like there may be some ways (companies) can train their model to be robust against attacks like that, but it’s really unclear what they are doing these days, because they don’t really talk too much about it, to see whether this is actually a really big concern to them or, if they have ways to circumvent it.
But once we deploy, once we start exploring a little bit more, perhaps we’ll see how these companies feel about it.
So I wouldn’t be surprised if they stopped — perhaps temporarily — scraping these data sets because they just have way too much data probably. But I think longer term, they kind of have to adapt their model, right? Your model can just be stuck in 2023 and at some point you need to learn something new. So I would say they probably will still keep scraping these websites and perhaps a little bit more carefully. But we don’t know at this point.
Jon Keegan is an investigative data journalist. This article was originally published on The Markup and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives license.
