Since it launched in earnest this spring, members of the Coral Project, a collaborative effort to rethink community engagement around news, have heard from all kinds of people. They’ve talked with everyone from trolls who plague comment sections to representatives from more than 125 different organizations in 25 different countries.

“What we’ve been doing up until this point is listening,” said Andrew Losowsky, who is leading the Coral Project, which is the result of a partnership between The Washington Post, The New York Times, the Knight Foundation, and the Mozilla Foundation.

Now, after months of listening, the 10-person Coral Project team is starting to build its first product: a tool for listening. The goal is to develop open-source software that will allow outlets to look at the communities surrounding their news organizations — from their comments sections to people who contribute user-generated content — and ascertain which users are the most trustworthy. Loswosky, along with Sydette Harry, the Coral Project’s community lead, and Greg Barber, director of digital news projects at The Washington Post, first announced the product last week at the annual Online News Association conference in Los Angeles.

Though the tool has yet to be named (“Internally, one of the coders is calling it ‘Trust-o-matic,’ but I don’t know if that will stick yet”), the Coral Project plans to finish building a beta version of the tool by the end of the year, then begin testing it with publishers in early 2016. In the next couple of weeks, it will also announce the four to six news organizations it’s partnering with to test the tool.

“What we’re building is a listening tool, and in order to build it we need to listen ourselves,” Losowsky said. “So the stage we’re at right now is identifying who those publishers are. We’re building out the framework within which these things live, but when we get to the specific levers to pull, we don’t know those yet.”

The Coral Project is still in the early stages of coding the product, so Losowsky couldn’t say exactly how it’ll work. But he did say the tool will ideally identify a few different metrics, such as the number of good or bad commenters, the number of repeat commenters, and which people would be interested in being contacted to help with future stories.

The readers who comment on articles are often among news sites’ most dedicated users, so it’s critical that outlets understand them and make their experiences better, Harry wrote in a post on the project’s website:

If journalism wants people’s eyes, their subscriptions, their time on sites, then people should have a way to contribute to that work, voluntarily and accessibly. Making journalistic platforms better places to comment means less living on Tweetdeck, chasing people who may or may not want you to use their content. It means connected, proven users whose behavior over time shows their desire to be involved.

These are your biggest fans, who will bring you the biggest return in exchange for feeling acknowledged, feeling heard. Worthwhile communities are worth the investment.

Comments in all their forms are not merely a problem.They are also proof that what you are producing touches our world. When they work well, they are the building blocks of what makes being online worthwhile. They are connected to the first letters to the editor, to social, and to what matters most: the people you write for and about. They are evidence that your content is part of the communities in which they live.

The Coral Project won’t host any of the data, and the tool will comply with each publication’s privacy policies, Losowsky said. The data the Coral Project collects will be anonymized logs of behaviors, like how many times users have replied to comments. The project may also do semantic analysis of the comments.

Ultimately, though, Losowsky said the Coral Project’s goal is to make the tool flexible enough for each outlet to use on its own terms. “In conversation with the publishers, we’ll get a sense of what different metrics will make sense for these different ideas of trust and reputation and then give the publishers the tools to shift those metrics in ways that make sense for them and get analytics out of that,” he said.

Once it tailors the tool for the publishers its working with, the Coral Project plans to make its work open source, so other outlets can also use its work and so that it will also live on beyond the duration of the project.

And with things such as an email newsletter, open Slack chats, and a public Trello board, the project has taken steps to publicize its thinking and get feedback from other interested parties.

The project is being funded by a two-year, $3.89 million grant from Knight. (Disclosure: Knight also funds Nieman Lab.) The Coral Project probably won’t turn its attention traditional comments until its second year, he said.

For now, the project is focused on its listening tool first, and then likely it’ll work on a tool to improve user-generated content contributions “so journalists can do very specific asks of the audience and to be able to connect that with comment identities,” Losowsky said. They are also looking at a way to provide commenters themselves better information about their contributions.

“Maybe [we can] turn around to them and say that ‘Our algorithm thinks that you sound kind of angry on Wednesday mornings’ — maybe that’s useful information,” he said.