Nieman Foundation at Harvard
The hosts of the most popular news podcasts hardly ever fly solo. A new report from the Pew Research Center finds news podcasts are more likely than other types of top podcasts to feature guests.
Only about 15% of the top-ranked podcasts — which Pew defined here as appearing in the top 200 of either Apple Podcasts or Spotify in 2022 — focus on news. But of those news-focused podcasts, an overwhelming majority (89%) featured guests in at least one episode, a higher share than podcasts that were not news-focused.
About a third of the most popular news podcasts regularly had guests (in 50% to 89% of their episodes). Another 13% of news podcasts almost always had guests (more than 90% of episodes with a guest) — compared to just 3% of podcasts that do not focus on news. Political podcasts were the only genre more likely to have guests in almost every episode. True crime podcasts, in contrast, were the least likely to bring on outside guests.
Top news podcasts often take on a similar format with the host interviewing a fellow journalist — typically a newsroom colleague — about their reporting. On Tuesday morning, NPR’s Up First had Liz Baker reporting on wild weather from California, White House correspondent Franco Ordoñez speaking to votes in Nevada, and higher education reporter Elissa Nadworny rehashing Dartmouth’s decision to start requiring SAT and ACT scores again. Over at The New York Times, The Daily had its Jerusalem bureau chief Patrick Kingsley on to discuss allegations about an important U.N. relief agency in Gaza. And The Journal podcast featured Wall Street Journal reporter Rebecca Elliott, who covers Tesla, on the “extreme blurring of friendship and fortune” between Elon Musk and his board of directors in its most recent episode.
Across all genres, podcasts that feature video content are more likely to feature interviews and guests than those who stayed in their audio-only lane. About 63% of all news-focused top podcasts have a video component, compared to 49% of other top-ranked podcasts.
One highlight of this report is buried in the methodology section, where researchers explain how they used OpenAI’s Generative Pre-Trained Transformer (GPT) 3.5 model to take on the tedious work of reading and coding thousands of podcast episode descriptions.
The podcast data set was something of a nightmare, as researchers Galen Stocking, Meltem Odabas, and Sam Bestvater outlined in a post published alongside their report on Tuesday. “The episode descriptions contain tens of thousands of names – not just hosts and guests, but also the names of people who are in the news or who are discussed but don’t appear on the show,” Stocking noted. “They are also not spelled or presented in a consistent format.”
Reading and coding 24,000 episode descriptions would typically “take a team of five workers about a month to go through the entire list if working eight hours a day, five days a week without any breaks,” the researchers estimated. Instead, their AI-powered model took just a few days.
“We’ve found that one of their biggest advantages is shortening the timeline for doing boring or rote classification work – exactly what we needed for this project,” Bestvater said.
The Pew researchers, naturally, were worried the large language model would “hallucinate” during its task. They took care while crafting their prompt — including instructing the model not to guess if it wasn’t sure if podcast episodes had guests or not.
They also randomly tested their model’s output against human researchers and found “it performed about as well as our own trained researchers did,” Odabas said. (According to the report, both achieved a little less than 90% accuracy. The model, like its human counterparts, sometimes classified similar names as the same person, or mistook host names for guests.) Unlike human researchers, however, the model sometimes refused to analyze episode descriptions with certain keywords.
“Our model’s content moderation guardrails sometimes rejected episode descriptions if they mentioned concepts like crime or sex, even in fairly generic terms,” Stocking said. “It took some time to figure out what was happening and how to work around it.”
Previously, the Pew Center deployed similar models to identify tweets either supporting or opposing the Black Lives Matter movement and to compare the language used in tweets by House Freedom Caucus members and other Republicans. You can read more about the report or about the researchers’ methods.
