German daily Die Welt’s internal grading system for its online articles sounds a little nerve-wracking. All published pieces are assigned a single score made up of five components and ranked; then a top 10 list is emailed out to the entire newsroom each morning, with some comments from an editor. But editors argue this isn’t some dystopian reduction of complex journalism into a single number. They say it’s about making totally transparent what the newsroom values in what it publishes online — not just clicks but also engagement time, not just views on the site but also how well a story travels on social media.

“The score was implemented mainly to increase the production quality of the product,” Oliver Michalsky, deputy editor of Die Welt online, told me over email. “Does the headline work well? Should we have incorporated a video? Is the story designed in such a way that readers read it until the end? Have we added sufficient links to further good content so that users are led to another WELT article? We also want to grow strongly in social networks. Therefore, the score requires reporters and editors to share the article. And a reporter gets an incentive to integrate video content.”

The five criteria that feed into the final “article score” are pageviews, time spent on the article page, video views, social shares, and bounce rate. An article can score up to 10 points for traffic, and up to five in the other four categories, for a maximum score of 30. This fall, when the site moves from a metered paywall model to a freemium one, a sixth category — how many subscribers did the article generate? — will be thrown into the mix. The data that feeds into the overall score comes from a mix of analytics tools including Chartbeat (individual reporters have access to Chartbeat also). Recently, the newsroom increased the number of clicks a story needs to be assigned the maximum score of 10 in the page views category from 20,000 to 30,000 per story, but the bump is “simply because we are currently developing very well in terms of reach.”

Not all traffic is good traffic, and a reader who comes to the site very briefly to read only that one article that has gone viral is not quite the audience Die Welt is going for, the organization explains in a post when it first implemented its article score system. In a Reuters Institute Report from February surveying how various news outlets are approaching analytics with specific goals in mind, the authors wrote of Die Welt’s system:

The development of the article score is part of a strategic process that focuses on making sure that Die Welt produces quality content that readers will find compelling enough not simply to click on and glance at, but actually to spend time with, perhaps share, maybe even pay for. It is a clear example of how Die Welt — and Axel Springer more widely — is developing forms of editorial analytics that are aligned with the specific editorial goals (quality content) and business model (metered paywall) of a specific organization. A tabloid like Bild (also an Axel Springer title), with a greater emphasis on advertising, high volumes of traffic, and a freemium pay model rather than metered paywall, calls for a different approach.

While “clicks” are still important in Die Welt’s article score system, they’re balanced out by four other measures, and reporters now have other tangible benchmarks to work towards.

This article, for instance, did fine in terms of pageviews, scoring 7 out of a maximum of 10) and ranking 26th on the pure traffic measure. But “due to its presentation and the inclusion of well-chosen images” — it’s an analysis of the body language of famous couples — “most users read it all the way to the end and also watched the video at the bottom of the article” (an impressive feat: you have to scroll for a pretty long time to get to the two-minute video). What the piece lacked in pageviews it made up in engagement and video views (scoring 5 in both), and ended up tied for No. 8 on the top 10 list:

Recaps of talk shows from the night before always do well online, according to Michalsky, but they were not often cracking the top 10 on the article score ranking, as a rights issue made it difficult for reporters to add relevant video to their pieces. So the paper started producing short video summaries of the people and issues discussed, using footage from other sources. “And it works!” he said, pointing to a recent story about criticism president of the European Union parliament Martin Schulz directed at Turkey and its president Tayyip Erdoğan on a talk show. The story ended up ranked second, hitting the maximum score for video views.

“My colleagues are very happy with the score because of its ‘fairness’ in comparison with pure reach,” Michalsky said when I asked how reporters were responding to this concrete feedback. Was there was any concern that certain topics might just perpetually generate more traffic than others, or whether some stories, such as longer Sunday features, would be sufficiently rewarded on the engagement measure? “We discuss articles regarding all these aspects at our newsroom meeting in the morning. For instance, we show the traffic difference between a successful and a weak headline for a specific article. And we give advice to team members on how to make use of social media for developing and distributing content.”

The topics covered in the top 10 are generally diverse: “It is the strength of the article score that not only stories on politics and the economy (the key high-reach sections of welt.de) lead the daily reach charts,” Michalsky said. “With the score, articles on culture or science topics suddenly are No. 1, even if they were only in 12th place in terms of reach. And our straightforward news stories actually achieve rather high engagement rates.”

Individual reporters are not tracked on how they’re performing overall in terms of article score, but reporters and editors can go into the CMS to find specific data points for all their daily articles there. Die Welt is also currently working on implementing a realtime score to help staffers who monitor the page.