Jonathan Soma: My reaction was, “Of course, this is what happens.” One-hundred percent of the time, when you have a chatbot, things are gonna go wrong. But when you have a lot of layers of bureaucracy in New York City and all the stuff that’s happening here at the moment, I think no one’s going to double check things. It was obvious that something like this was going to happen at some point, and it was just a matter of when and where.

But terrifying. You have a tool that is pitched as a useful tool to help people live their lives in a way that is most effective to obeying the law. Then it’s pitching them lies under the banner of, “This is sourced, this is true. You can trust this.” That’s the big problem with chatbots — they will cite sources, they will say facts that are not facts. Then maybe there’s a little asterisk down at the bottom that says,”Sorry, some things here might not be true, double check all of them.” Which is just absurd.

Soma: On the technical end, right now, chatbots that generate responses to questions — so retrieval-augmented generation — is the hottest thing. It is what everyone wants to do, mostly because they want to replace employees who are doing customer service work.

It is very easy to get a small proof of concept going. No matter what tool they used to generate this, I know exactly how it worked, there’s no mystery there. The mystery in these chatbots is how you can get them to be accurate. And it’s still a big problem that all of these chatbots have, and clearly this one as well.

Soma: Yes. So I have been using these tools for quite a while. One of my favorite projects that I’ve made was a chatbot that would answer questions about Hungarian folktales, based on a book that was in Hungarian. I was using it to teach investigative journalists about how you can do investigations across different languages, because these chatbots can read different languages and interact with things that maybe you can’t speak.

It was very fun, again, as a proof of concept. But if someone was going to sit me down and make me defend a PhD thesis based on a Hungarian folktales chatbot, I don’t think I would pass.

Soma: What happens is you take a list of documents, and you feed them into a machine. Every, let’s say paragraph or two of that document, is assigned a series of numbers that describes this as the meaning or the topic of that paragraph. When someone asks a question, what happens is the machine takes that question, converts it into the same sort of numbers, then finds maybe five or 10 sets of paragraphs that are close to the number that your question is. So in theory, it’s all the texts that are similar to your question, probably relevant to generating an answer. And then it sends the question along with all of those documents to the chatbot or to GPT, to the AI, and say, “Please answer this question based on these five or 10 sets of paragraphs that we think are relevant to the question.”

I did that by running some code that assigned all the numbers and cut all the paragraphs and sent everything out to GPT to get that answer. There are probably hundreds of tools that do this. It’s very, very common. A lot of slight variations on the approach, but generally speaking, you’re finding relevant documents and sending them to the AI along with the questions.

Soma: Yes. So I went online, and I just found a few different sources that were on the various New York City nyc.gov websites that might have relevant information about this — the fact sheets that you all cited about the true answers to these things. And yes, it seemed to work fine whenever I asked those questions.

Soma: If I were being gracious, I would say that there is always the ability of the AI to make things up and hallucinate. Additionally, if you have a very large set of documents, it might be difficult to find the ones that are actually relevant. Because that’s an automated process, and maybe when I say, “Can I take a portion of my workers’ tips?” it finds documents that might not be relevant. And then the chatbot isn’t able to answer because the documents aren’t necessarily relevant.

But there were so many basic, basic questions that were answered incorrectly, I can’t imagine the documents were actually loaded. There must just have been gross malfeasance in terms of setting this chatbot up. Because it’s not hard to have it be right most of the time. The issue was getting it to be right 100 percent of the time. And from what I can tell, this city-based chatbot was almost never right, which is hard to do. It’s hard to be that wrong.

Soma: Yeah, there are so many different opportunities for things to go wrong when you’re trying to use a chatbot that’s related to a set of documents. The AI can make things up. It can be referencing out-of-date documents, it can be referencing irrelevant documents. Maybe the way that you ask the question isn’t the way that the AI interprets the question, and so it’s answering something that is similar, but not exactly the same.

It is 100% guaranteed that, at some point, there’s going to be some sort of mistake in that chain, and there’s going to be some sort of error introduced, and you’re going to get a wrong answer. There’s almost no way to fact-check it. There are ways to make it slightly better and slightly better and slightly better and slightly better. But getting to 100%, I would say, is almost impossible.

Soma: Chatbots are great for low-stakes things. They are great when something is fun, they are great for a task where you do not need 100 percent accuracy, when you just want a little bit of guidance. “What restaurant should I go to?” Things like that. But relying on it for something like legal advice, based on what you can do in the city, is pretty irresponsible.

(After publication of The Markup’s story, several disclaimers were added to the bot, including one cautioning users to “not use its responses as legal or professional advice.”)

Soma: With AI-related tasks, there’s always going to be margin of error. You have to use it for tasks where it’s OK if there is an error. For example, if there is something where I might have an intern do a task, I might say, “OK, I’m an expert at this, I could do it. But it’s probably a better use of the intern’s time, and maybe sometimes they’re going to make a mistake. I’m going to build into my process the idea that I need to double-check this work. I need to make sure that there are no errors, because, you know, sometimes something might go wrong.”

When you’re using AI tools, you just need to test all of the time. It’s very easy to get into a conversation with AI and think, “Wow, this is amazing. I’m getting an incredible poem about my cats. I’m getting a recipe about this thing or the other thing. I’m getting feedback or edits on something I have written,” and it’s very easy for you to say, “Yes, I will accept this. No, I won’t accept that edit. Yes, this poem is adorable, or no, write another one about how the cat is mean bean instead of the cat is nice,” or whatever. So you just need to make sure that sort of capacity for errors is built in. Either you’re reviewing everything manually, or it’s a task where instead of doing one review of a piece of legislation, instead you’re looking at 100,000 pieces of legislation, or a million pieces of legislation, something that you as a human being couldn’t do. And you say, “OK, AI, you go do it, I understand that maybe 3 percent of the time, you’re gonna give me some incorrect information. But because I’m doing something at scale, maybe it’s okay that there’s errors here and there. All I want is the general gist to be true, or the general gist to be accurate.” But you can’t do that when you’re building a chatbot and every single conversation has to be meaningful, and has to be accurate for the person who is using it.

Soma: I think that AI in general is absolutely useful for journalism, and I’ve been teaching machine learning and AI to journalists long before ChatGPT hit the scene. I think it is explicitly chatbots that are probably the most problematic part, because they are so confident in everything that they say.

But journalists as a whole are very used to talking to people who are very confident, making statements that need to be fact-checked, that need to be cross-referenced. The step that we need to make as a society is moving from, “This came from a machine, it must be correct,” to, if I’m talking to a friend of mine who says something crazy, “I need to double check that, I need to cross reference it to make sure that it is accurate.”