How journalists decide when to trust numbers

Numerical information is a central piece of journalism. Just look at how often stories rely on quantitative data — from Covid case numbers to public opinion polling to economics statistics — as their evidentiary backbone. The rise of data journalism, with its slick visualizations and interactives, has reinforced the role and influence of numbers in the news.

But, as B.T. Lawson reminds us in a new article in Journalism Practice, though we have plenty of research on this decade-long boom in data journalism, much of the research “overstates the significance of the data journalist within the news media. Yes, data journalists are now a mainstay of most news organizations, but they are not the only journalists using numbers. Far from it.”

Indeed, in contrast to the 1960s and 70s era of computer-assisted reporting, when a small minority of specialized reporters worked with data but most reporters did not, nowadays virtually all journalists are expected to engage with numbers as part of their work. Which brings up a potential problem: Some research suggests that journalists rarely challenge the numbers they receive, leading them to accept and reproduce the discourse around those numbers from their sources.

To get a clearer picture of how journalists draw on numbers and narratives about them, Lawson examined reporters’ use of numbers in their coverage of seven humanitarian crises in 2017. The author did this in two ways: first through a content analysis of 978 news articles from U.K. news media (to look for some direct or indirect form of challenging statistics, cross-verifying one claim relative to another, etc.), and then through interviews with 16 journalists involved in at least one of those stories, to gain additional insights into the process of receiving and reporting on numbers.

The title of the resulting article — “Hiding Behind Databases, Institutions and Actors: How Journalists Use Statistics in Reporting Humanitarian Crises” — indicates something about one of its findings: namely, that journalists covering humanitarian crises rely heavily on numbers, often provided by NGOs or the UN, but they seldom verify the numbers they use, mainly because they see it as outside their role to do such work and because they “hide behind” the perceived credibility of their sources.

Instead, Lawson writes, “when it comes to verifying numbers in reporting humanitarian crises, journalists perceive their role to be limited to the assessment of trustworthy sources rather than the direct interrogation of the number itself” (emphasis added). So, journalists were found to develop, with remarkable consistency across media organizations, a practice of gathering “evidence of evidence.” This was a way of determining which people and institutions they believed they could trust, based on three criteria: a group’s track record with accuracy, how much it engaged in advocacy work, and whether the journalist had personal experience working with that source “on the ground” in some way.

But more than any NGO or other institution, it was public databases that journalists believed to be the most trustworthy source of numerical information. Reporters imagined such databases — provided by groups such as the OECD — as apolitical (non-controversial) and rational (non-emotional).

What happens, then, when something about the data is wrong? “How do journalists maintain their credibility,” Lawson asks, “when a number turns out to be inaccurate, unreliable or misleading?”

This study suggests that journalists hide behind their sources to protect themselves from the threat of criticism. This happens in two stages: “if the statistic turns out to be misleading, [journalists] can refer to the trustworthy institutional source, and, if that is not enough, they can point to the quantitative realism of the database from which the statistic was derived.”

Finally, a sobering finding emerged from the interviews. Lawson found that in their use of numbers to cover humanitarian crises, journalists rarely relied on the traditional “two-source rule” of journalism; instead, presumably because they were pressed for time, reporters “did not speak of checking any facts and mainly relied on one source. Therefore, the ‘actual’ checking of numbers was almost always inexistent, replaced almost entirely by the ‘evidence of evidence’ approach.”

Research roundup

“Terrorist organizations in the news: A computational approach to measure media attention towards terrorism.” By Lea Hellmueller, Valerie Hase, and Peggy Lindner, in Mass Communication and Society.

“What is terrorism (according to the news)? How the German press selectively labels political violence as ‘terrorism.'” By Valerie Hase, in Journalism.

The news media’s coverage of terrorism has been extensively examined by scholars across a variety of platforms and lenses. Given the ground that has already been covered, it’s notable when two large-scale studies that push our understanding of the issue forward land in the same month. Valerie Hase of the University of Zurich is an author on both studies, but they cover different data sets — one on U.S. and U.K. media, and the other on German news — as well as different questions.

In the first study, Hellmueller and her colleagues examined what factors lead to greater media coverage of terrorist attacks. They looked at American and British news between 2014 and 2016 and found an inordinate, almost exclusive, focus on the Islamic State. They also found that greater fatalities and attacks on civilians and tourists led to greater media attention. Continuity — i.e., whether the media organization had covered the perpetrating group previously — was also a significant factor. The coverage dynamics they found create a conundrum for journalists: Larger and more violent attacks naturally are seen as deserving more coverage, but since terrorist groups are seeking media coverage, those patterns actually encourage more brutal attacks.

In the second study, Hase looked at when the German press chooses to label acts of political violence as “terrorism.” Using a set of more than 5,000 German articles about violent incidents between 2012 and 2018, Hase compared various factors as predictors of the incidents being labeled as terrorism.

She found that the terrorism label was applied not based on the incident’s victims (there was no difference if civilians or tourists were targeted as opposed to combatants), but based much more on its location (in Western countries) and its perpetrator (Islamic extremists as opposed to nationalist or left- or right-wing extremists; groups as opposed to individuals; attacks on perpetrators’ own soil as opposed to international incidents). Hase posits that journalists’ national identity comes to the fore as a factor explaining when and why they choose to describe attacks as terrorism, especially when in-group members are threatened by out-group members.

“Changing journalistic information-gathering practices? Reliability in everyday information gathering in high-speed newsrooms.” By Els Diekerhof, in Journalism Practice.

Journalists have been talking about the tension between immediacy and accuracy (as we often hear it, “getting it first” vs. “getting it right”) since time immemorial. The conventional wisdom is that journalists are getting it right less often during the digital journalism age because an accelerating news cycle has made getting it first an overwhelming priority — and there’s some good evidence for parts of this idea.

Diekerhof took a closer look at this relationship between immediacy and accuracy by making detailed observations of journalists at eight high-speed Dutch newsrooms. She found that the immediacy/accuracy tension isn’t quite the dichotomy we’ve thought of it as. In the everyday information gathering she observed, the speed of journalists’ work didn’t hinder its accuracy because most of the basic information for the story had already been gathered by the time it came to them, through wire services, news feeds, and social media. Their information gathering was primarily additional information gathering — adding context, checking facts, filling out additional facts — that was less factually precarious than developing an original story.

In these cases, Diekerhof argued, journalists were working quickly because their verification work was simple: “Information practices in high-speed newsrooms are not characterized by hard to find and complicated information.” They certainly cared about getting it right; they were just able to do that in a way that fulfilled their organizations’ demand for speed because they were building on information others had already gathered. It’s a shift in newswork that we’ve seen across journalism, but Diekerhof argues it’s perhaps not as much of a threat to the reliability of news as we might think.

“Data journalism in favela: Made by, for, and about forgotten and marginalized communities.” By Mathias-Felipe de-Lima-Santos and Lucia Mesquita, in Journalism Practice.

A mountain of research has been produced on data journalism over the past eight years or so, but as some scholars have noted and is so often the case in our field, that research has been disproportionately focused on Western and often wealthy societies. But there is, of course, a robust practice of data journalism in a variety of contexts in the Global South, and this study looks at a particularly interesting context: the marginalized communities of Brazil’s favelas.

As the authors note, official data about marginalized people such as those living in the favelas is scarce, and its use (and non-use) often serves to perpetuate historical inequities. De-Lima-Santos and Mesquita examined three organizations dedicated to journalism for and about these communities, and found four strategies that allowed these organizations to produce data stories despite these challenges. The first was citizen participation, in which journalists used technological tools (like WhatsApp and Google Forms) to help marginalized groups produce their own data to fill in the gaps in official data.

The second and third strategies were activism and collaboration, which involved pressing public officials for greater data access and better representation, and working with civic tech organizations and foundations to support their novel approaches. Finally, they worked to humanize their data and present it in ways that connected with their communities, including radio and sound-equipped cars for those without smartphones. These strategies together, the authors argued, served to de-Westernize their data journalism practice and turn it into a tool to advocate for an underserved public.

“Appreciating news algorithms: Examining audiences’ perceptions to different news selection mechanisms.” By Glen Joris, Frederik De Grove, Kristin Van Damme, and Lieven De Marez, in Digital Journalism.

The role of algorithmic recommendation systems in feeding people an endless stream of like-minded information is one of those concerns that has taken hold not just among scholars or media professionals, but among ordinary media consumers as well. Tell people that you work in (or study) media, and you’ll likely hear a complaint at some point about how our news feeds and apps keep shoveling more and more of whatever viewpoints we’ve expressed an interest in at us.

It’s easy to say we’re sick of having algorithms recommend more of the same news to us, but do we actually want anything different? That’s the question driving this study. Joris and his colleagues surveyed Belgian adults to find out about their preferences among three types of news recommendation systems: content-based similarity (more news like what you’ve already read), collaborative similarity (news like what’s popular among your friends or other users), and content-based diversity (news from viewpoints or on topics you don’t usually read).

Content-based similarity came out the clear favorite, which probably shouldn’t surprise us, though the researchers did find some possible affinity for a more personalized and carefully selected news diversity. They also looked at factors influencing overall openness to algorithmic news recommendation systems and found that the strongest influence was a feeling of information overload, more than any demographic factors.

“Local newspapers’ transition to online publishing and video use: Experiences from Norway.” By Roel Puijk, Eli Beate Hestnes, Simon Holm, Andrea Jakobsen, and Marianne Myrdal, in Journalism Studies.

Researchers have been writing about local newspapers’ transitions to digital media for two decades now, with distinct phases in the heady early days of the late 1990s and early 2000s and the mid- and post-apocalyptic environment of the late 2000s and early 2010s. Puijk and his co-authors documented an intriguing evolution in this process: Small local newspapers who have been online since the 2000s, but only begin focusing their attention on the web in the mid-2010s, as their papers abandoned the free-content model for a paywall.

As a grad school class project, Puijk and his students interviewed editors at five newspapers in a single Norwegian county. They focused on the role of video in particular, and found a sharp turn from the more TV-style multiplatform video efforts of the early 2010s toward a heavy emphasis on live streaming video.

The shortform, story-based video was tied to an advertising-based model and didn’t hold much value in the world of digital subscriptions. (It also required a level of craft that couldn’t be sustained at these papers.) But the popularity of live streaming video, especially local sports, in driving subscriptions, has made local broadcast rights especially valuable for these Norwegian papers. The upshot of those video changes, along with others, is greater local differentiation, even among newspapers within the same conglomerate.