All of The Platforms™, whether search or social, face issues about surfacing vaccine mis- and disinformation (see: Pinterest’s efforts), and we’re supposed to be filling the data voids with helpful, accurate information to counter the anti-vaxxers. But it’s not only on the conspiracy edges where the media runs into trouble covering health news and research.

Less than a third of news stories about health findings sufficiently discussed cost, and less than 40 percent mentioned potential harms or the quality of the evidence, according to a review of more than 2,600 health news articles by HealthNewsReview, a journalistically-driven site that analyzed health reporting. (Don’t even get started on the press releases.) Ten percent of the articles received a one or zero star rating on HealthNewsReview’s scorecard, with 14 percent hitting a perfect five stars.

What does that look like? A one-star story, for example, is The Guardian’s “Virtual reality to help detect early risk of Alzheimer’s.” “That is misleading. The researchers haven’t even recruited study participants yet, much less conducted the study or analyzed the results,” wrote a trio of reviewers, including a university’s public information officer/freelance science writer, medical director of a Medicaid Health plan (yes, with a medical degree), and HealthNewsReview’s own deputy managing editor, each of whom signed an industry disclosure agreement. “We can understand that a research institution may want to raise awareness of a forthcoming study in order to recruit study participants. We are less clear on why a major news outlet would write about the research — which hasn’t even taken place yet — as if it is already newsworthy and generating data that clinical decisions can be based upon.”

That was the last review that HealthNewsReview published. Battling funding loss as a nonprofit and an impasse of how to most constructively criticize news outlets, HealthNewsReview ceased operations in December 2018, amid an environment where YouTube’s recommendations coax vaccine-searching users to anti-vax videos. The 13-year-old site’s archives are still available, but probably only for the next three years or so, according to Gary Schwitzer, HealthNewsReview’s founder and publisher.

Schwitzer used this metaphor, talking with me from a Las Vegas-area Costco parking lot on an RV trip with his wife: “I’m looking at the snow-capped mountains. They’re beautiful and wonderful, but the peaks of excellence are far too occasional and the valleys are the drumbeat of dreck [that] may overcome the good…The dreck is conflicted single-source stories that should not be called journalism but should be PR and advertising drivel, because it is simply taking spoon-feeding in an unquestioning way and fawning over it and not showing any connection with the needs of a desperate healthcare population that we could be helping,” he said. “My heart soared with some of the things from ProPublica, Kaiser Health News, and Stat, but Stat is among the ones we also have criticized. Once again, today I get their morning newsletter and today it was sponsored by the biotech industry.”

For most of HealthNewsReview’s existence, Schwitzer led the crusade to critique health reporting as a one-man band funded by foundations. He focused on conflicts of interest in health journalism and the impact of misleading media reports on patients, accruing 2,870 blog posts, 50 podcast episodes, more than 50 editorial contributors (most with a MD, PhD, or MPH after their name), and six-ish full-time staffers. Its weekly email had 6,000 subscribers. HealthNewsReview arose from his frustrations working in local TV in Milwaukee and Dallas and then heading the medical news unit at CNN. He was inspired by a project called Media Doctor Australia, which used a standardized tool to evaluate stories. Here’s how Schwitzer described the situation for Boing Boing three years after launch, back in 2009:

It was mostly the CNN experience that frustrated me. In the early days of CNN, we had this tremendous, exciting opportunity. The channel could be the place to go in-depth with background and be analytical and contextual. But then the management side swung the other way and preferred to be the wire service of the air — take anything happening anywhere and report it with a quick turnaround. That’s the continued recipe for disaster in my eyes. Into that was thrown the maelstrom of artificial heart experiments in the early 1980s. I saw how all of the incentives were to just have the information everyone else had, but more often. It got to be hourly briefings on patient urine output and stuff like that, rather than reporting on evidence and tech assessment, and cost, and access and all the things that now become our criteria on HealthNewsReview.

And HealthNewsReview did establish a rigorous system for guiding reporters in healthcare news and scoring the pieces that came through. The site shares its 10 must-have elements for an article or press release to be acceptable (reporters, take note):

1. Does the story adequately discuss the costs of the intervention?
2. Does the story adequately quantify the benefits of the intervention?
3. Does the story adequately explain/quantify the harms of the intervention?
4. Does the story seem to grasp the quality of the evidence?
5. Doe the story commit disease-mongering?
6. Does the story use independent sources and identify conflicts of interest?
7. Does the story compare the new approach with existing alternatives?
8. Does the story establish the availability of the treatment/test/product/procedure?
9. Does the story establish the true novelty of the approach?
10. Does the story appear to rely solely or largely on a news release?

But of course, rating a story post-publication doesn’t always translate to a better story the next time around. The site also struggled to straddle the line of constructive criticism in a way that rank-and-file journalists and editorial leadership accepted. (Yes, we are critiquing a critiquer. We are meta.) The low honey-to-vinegar ratio didn’t endear the site to some journalists. Undark described some of the site’s pitfalls in a feature:

“I will not mourn the disappearance of HealthNewsReview,” said Maggie Fox, a senior health writer at NBC News, in an interview with Undark. “I don’t think they ever did the job well,” she said, describing the site’s tone as “mean-spirited.”

Fox did stress the value of media criticism in general. But, she argued, HNR’s approach had done too little to engage with journalists about their work — by not seeking comment, for example, before publishing criticisms — and its criteria were not realistic for all stories in all newsrooms. “Their demands are for very long, very wordy stories with a lot of background information in them, without taking into account that many organizations have word count limits, and that understandability for whoever your audience is,” Fox told me, “you might not need or want all that detail in there.”

Both in his quotes in that Undark piece and in our parking-lot interview, Schwitzer stuck to his guns. He said he knows he “probably did sign our death sentence” by insisting the site not cut back on its resource-intensive three-reviewer system. The site also relied heavily on philanthropic dollars that ultimately ran dry. The Foundation for Informed Medical Decision Making supplied Schwitzer with grants, but that organization eventually became the Informed Medical Decisions Foundation and then merged with a nonprofit patient content company called Healthwise — which then pulled the funding in 2013. A private foundation came through with a $1.3 million grant in 2014, but that also eventually came to an end.

Schwitzer pushed for foundations to support HealthNewsReview but kept hitting the gap between healthcare foundations and journalism foundations in an era of emerging journalism philanthropy. He also had little faith in crowdfunding to hit the kind of level the grants provided to keep the site’s system afloat. And industry advertising or corporate contributions are a no-go: “I always said and stood by this: I would rather shut it down than take industry money, for all the same reasons that run so deep in me,” he said.

So which news organizations did the best job reporting on health issues? The overall winner by a healthy margin was Vox, which earned an average of 4.46 stars over 26 reviews. (Notably, Vox has no qualms about going long on health stories, and it has none of the constraints of print space or broadcast time.) Here’s what reviewers said about a Vox story by German Lopez last year about an analysis of alternatives to 12-step programs in addiction treatment:

This is excellent coverage of a study suggesting that Alcoholics Anonymous (AA) and similar 12-step programs aren’t the only worthwhile support groups for those recovering from alcohol use disorders. Alternative programs that (unlike AA) aren’t based on a spiritual philosophy may be just as effective, the results show.

The story earns high marks for thoroughly describing the evidence that forms the basis for the coverage and explaining its key limitations. It also features perspective from an independent expert and notes the big gap in availability between AA and its less established counterparts.

But do these programs have any downsides and what do they cost? Readers would have benefited from hearing those specifics, but the omissions are relatively minor considering the broad scope of the coverage.

Next up in the site’s final rankings were Stat (3.91 stars over 44 reviews), The Philadelphia Inquirer (3.69 stars over 75 reviews), and The Associated Press (3.66 stars over 336 reviews).

And the worst? The major broadcast TV networks all fared poorly. (though the site was evaluating their online stories, not actual on-air segments). A few recent reviews illustrate what the site regularly dinged stories for; here’s ABC News:

ABC News pursues a story idea they heard about in a special issue of National Geographic magazine and calls it a “breakthrough fertility test.” But what ABC delivers is one satisfied success story. We’re given no broader context other than the vague reference to the fact that it hasn’t been tried “on many” women. How many? And with what results? How can the audience judge whether the breakthrough label fits in this case? They can’t.

Yes, couples with infertility problems are desperate for answers. But that is why context and evidence is so important — not merely single anecdote success stories that may not be representative of the experiences of other women who have been similarly treated.

NBC News:

This story suggested that lingering fear from the 1990s fen-phen scare continues to drive patients away from the “latest generation” of weight-loss drugs, which, it asserted, could safely help some people with obesity lose enough weight to improve their risk factors for heart disease.

But the story didn’t give readers much information to go on. There was no data quantifying the benefits of drugs now on the market or comparing them with alternatives, such as surgery. The story also didn’t mention cost or potential harms and relied largely on quotes from two physicians who have received significant compensation from companies that make weight-loss drugs–which wasn’t disclosed in the story. It is vital that journalists disclose such details.

And CBS News:

This CBS news story describes a possible mechanism by which botox (onabotulinumtoxinA) treatments could be an effective treatment for major depression. It cites a dermatologist who explains the possibility of a feedback loop between the facial muscles that express emotion and the amygdala in the brain. It suggests that botox may offer hope for treating not only depression, but also social anxiety and bipolar disorder.

However, the only evidence this story provides is a single anecdote by an army veteran. The only expert it cites is a dermatologist who stands to gain financially if the treatment receives FDA approval, and who has been a paid consultant for the company manufacturing the drug.

The Guardian — the source of that story about the results of a VR study that hasn’t even been done yet — doesn’t look good in HealthNewsReview’s analysis, with a meager 2.29 stars over 42 reviews. It was regularly knocked for “rehashed” press releases and “spoon-fed” stories.

And the worst of all? Fox News, with just 2.13 stars across 40 stories. (“Fox News regurgitates a hot mess from the UK Sun”; “Classic disease-mongering”; “Yet another single-source Fox News story that reads like sponsored content”; “No original reporting: Fox News rehashes news release”; “news via lone anecdote.”)

Schwitzer still holds out hope for non-industry subsidized health news reviewing: “The $1.6 million I had over two years is a drop in the hat when you look at the healthcare advertising and marketing budgets of any of the single players in the industry. That money is out there. Maybe somebody else can make the pitch better than I did.”