Narayanan: Sayash Kapoor and I call it a bullshit generator, as have others as well. We mean this not in a normative sense but in a relatively precise sense. We mean that it is trained to produce plausible text. It is very good at being persuasive, but it’s not trained to produce true statements. It often produces true statements as a side effect of being plausible and persuasive, but that is not the goal.

This actually matches what the philosopher Harry Frankfurt has called bullshit, which is speech that is intended to persuade without regard for the truth. A human bullshitter doesn’t care if what they’re saying is true or not; they have certain ends in mind. As long as they persuade, those ends are met. Effectively, that is what ChatGPT is doing. It is trying to be persuasive, and it has no way to know for sure whether the statements it makes are true or not.

Narayanan: There are very clear, dangerous cases of misinformation we need to be worried about. For example, people using it as a learning tool and accidentally learning wrong information, or students writing essays using ChatGPT when they’re assigned homework. I learned recently that CNET has been, for several months now, using these generative AI tools to write articles. Even though they claimed that the human editors had rigorously fact-checked them, it turns out that’s not been the case. CNET has been publishing articles written by AI without proper disclosure, as many as 75 articles, and some turned out to have errors that a human writer would most likely not have made. This was not a case of malice, but this is the kind of danger that we should be more worried about where people are turning to it because of the practical constraints they face. When you combine that with the fact that the tool doesn’t have a good notion of truth, it’s a recipe for disaster.

Narayanan: ChatGPT is part of the generative AI category. Technologically, it’s pretty similar to text-to-image models, like DALL-E [which creates images based on text instructions from a user]. They are related to AI that’s used for perception tasks. This type of AI uses what’s called deep learning models. About a decade ago, computer vision technologies started to get good at distinguishing between a cat and a dog, something people can do very easily.

What’s been different in the last five years is that, because of a new technology called transformers and other related technologies, computers have gotten good at reversing the perception task of identifying a cat or dog. This means that, given text prompts, they can actually generate a plausible image of a cat or a dog or even fanciful things like an astronaut riding a horse. The same thing is happening with text: Not only are models taking a piece of text and classifying it, but given a prompt, these models can essentially run classification in reverse and produce plausible text that might fit into the category given.

Narayanan: I think the best example of automating judgment is content moderation on social media. It is clearly imperfect; there have been so many notable failures of content moderation, many with deadly consequences. Social media has been used to incite violence, even perhaps genocidal violence in many parts of the world, including in Myanmar, Sri Lanka, and Ethiopia. These were all failures of content moderation, including content moderation AI.

However things are improving. It is possible, at least to some degree, to take the work of human content moderators and train models to make those judgments about whether an image represents nudity or hate speech. There will always be inherent limitations, but content moderation is a dreadful job. It’s a job that’s filled with the trauma of looking at images of gore and beheadings and all kinds of horrible things day in and day out. If AI can minimize the human labor, that’s a good thing.

I think there are certain aspects of the content moderation process that should not be automated. Deciding the line between acceptable and unacceptable speech is time-consuming. It’s messy. It needs to involve input from civil society. It’s constantly shifting and culture-specific. And it needs to be done for every possible type of speech. Because of all that, AI has no role here.

Narayanan: This is the kind of AI where decision-makers predict what someone might do in the future and use that to make decisions about them, often to preclude certain paths. It’s used in hiring, it’s famously used in criminal-risk prediction. It’s also used in contexts where the intent is to help someone. For example, this person is at risk of dropping out of college; let’s intervene and suggest that they switch to a different major.

What all of these have in common is statistical predictions based on rough patterns and correlations in the data about what a person might do. These predictions are then used to some degree to make decisions about them, and in many cases, deny them certain opportunities, limit their autonomy, and take away the opportunity for them to prove themselves and show they’re not defined by statistical patterns. There are many fundamental reasons why we might want to consider most of these AI applications to be illegitimate and morally impermissible.

When an intervention is made based on a prediction, we need to ask, “Is that the best decision we can make? Or is the best decision one that doesn’t correspond to a prediction at all?” For instance, in the criminal-risk prediction scenario, the decision that we make based on predictions is to deny bail or parole, but if we move out of the predictive setting, we might ask, “What is the best way to rehabilitate this person into society and decrease the chance that they will commit another crime?” It opens up the possibility of a much wider set of interventions.

Narayanan: Assume that some of the wildest predictions about ChatGPT are true and it will automate entire job categories. By way of analogy, think about the most profound information technology developments of the last few decades, like the internet and smartphones. They have reshaped entire industries, but we’ve learned to live with them. Some jobs have gotten more efficient. Some jobs have been automated, so people have retrained themselves, or shifted careers. There are some harmful effects of these technologies, but we’re learning to regulate them.

Even with something as profound as the internet or search engines or smartphones, it’s turned out to be an adaptation, where we maximize the benefits and try to minimize the risks, rather than some kind of revolution. I don’t think large language models are even on that scale. There can potentially be massive shifts, benefits, and risks in many industries, but I cannot see a scenario where this is a “sky is falling” kind of issue.