Sunday afternoon, Politico came out with a “special report” on the relationship between the strength of local newspapers and support for Donald Trump in the 2016 election. “Trump thrives in areas that lack traditional news outlets,” the headline said:

An extensive review of subscription data and election results shows that Trump outperformed the previous Republican nominee, Mitt Romney, in counties with the lowest numbers of news subscribers, but didn’t do nearly as well in areas with heavier circulation.

Politico’s findings — which put Trump’s escalating attacks on the media in a new context — were drawn from a comparison of election results and subscription information from the Alliance for Audited Media, an industry group that verifies print and digital circulation for advertisers. The findings cover more than 1,000 mainstream news publications in more than 2,900 counties out of 3,100 nationwide from every state except Alaska, which does not hold elections at the county level…

That gives new force to the widely voiced concerns of news-industry professionals and academicians about Trump’s ability to make bold assertions about crime rates, unemployment and other verifiable facts without any independent checks. Those concerns, which initially were raised during the campaign, were largely based on anecdotes and observations. Politico’s analysis suggests that Trump did, indeed, do worse overall in places where independent media could check his claims.

I have a lot of sympathy for those who argue that the decline of local newspapers has led to a decline in government accountability, political engagement, and a host of other democratic ills. There’s been a fair amount of academic research supporting the claim; I’ve always liked this 2009 Princeton study that used a nifty natural experiment to show that after the closure of the Cincinnati Post in 2007, “fewer candidates ran for municipal office in the suburbs most reliant on the Post, incumbents became more likely to win re-election, and voter turnout fell.” I’ve long written about the platform-worsened decline of local news and the alarming concentration of our journalistic resources in New York and Washington.

But this study doesn’t really show what people are claiming it does. The underlying argument might be true, but the data the authors are using doesn’t prove their case. Here are a few of the issues I have with it.

The study’s definition of a “news desert” is awfully strained. “News desert” is a term without a universally agreed-upon definition — but there are a number of problems with the one Politico uses.

First, to the extent there’s been any consensus, it’s generally that a news desert is a place with no local news outlets at all. (CJR: communities “with no daily local news outlet at all”; UNC: communities that “lose their local newspapers.”) Others go with something looser like “places where it is difficult to access daily, local news and information” or even “a community overlooked, if not entirely ignored, by the media.”

But all of these definitions are about news supply. Politico’s definition is all about news demand — what percentage of people in a county subscribe to a newspaper. The demand side of news is a real problem! But low audience demand doesn’t make a place a “news desert.”

The study also seems to arbitrarily decide that the bottom 10 percent of American counties in newspaper circ are by definition “news deserts.” There’s no particular defense offered of that cutoff. And again, there’s no evidence offered that these places are really “deserts.”

Finally, Politico may be surprised to learn that local TV and radio stations exist. Local newspapers are a minority interest just about everywhere in America. Pew data from 2016 found that print newspapers were a distant fourth in how Americans got their news (57 percent for TV, 38 percent for online, 25 percent for radio, and 20 percent for print newspapers). It trails TV badly in every age group. (I’m from a small country town of 6,000 people and I can tell you local radio stations were at least as important as the town weekly for community news.)

What about during the 2016 election — maybe tons of people were getting their Trump and Clinton news from the local daily? Nope — when asked what was their main source of news about the 2016 election, only 3 percent said it was a local newspaper.

News deserts are a real thing — but you can’t just decide that a single (and extremely flawed, as we’ll see in a bit) data point about America’s fourth-most-popular form of media is the be-all and end-all.

The vast majority of local newspapers play only a tiny role in how voters get information about a presidential election. Here’s Politico editor Carrie Budoff Brown:

New: A Politico analysis found a clear correlation between low news subscription rates and Trump’s success against Clinton and when compared to Romney in 2012. Trump did, indeed, do worse overall in places where independent media could check his claims. https://t.co/WKezgzXeDN

— Carrie Budoff Brown (@cbudoffbrown) April 8, 2018

Come on. Does anyone think that the job of “independent media” checking Trump’s claims really fell in any significant way to local newspapers? It was up to The Poughkeepsie Journal to fact-check him?

If this was a study about local congressional or legislative races, it might make sense to look at local media. But the presidential race has been the dominant turf of national media for literally decades. In that Pew study, only 11 percent of Americans cited any local news source — TV, radio, or newspapers — as their main source of election news.

Then there are lines in the story like this:

Politico’s analysis shows how [Trump] succeeded in avoiding mainstream outlets, and turned that into a winning strategy: Voters in so-called news deserts — places with minimal newspaper subscriptions, print or online — went for him in higher-than-expected numbers.

It is just ahistorical to say Trump’s campaign “succeeded in avoiding mainstream outlets.” Candidate Trump gave tons of interviews throughout the primaries! And when he scaled those back in the general, one of his strategic shifts was shifting from national to local media! Whatever perfectly legitimate criticisms you want to make about how the media covered Trump, to say Trump “avoided” coverage in mainstream outlets is just…wrong.

They’re rediscovering the rural/urban divide. Cities and suburbs have more people, higher subscription rates, and more Hillary Clinton voters. Rural areas have fewer people, lower subscription rates, and more Donald Trump voters. Given those facts, it’s not that surprising that Politico would find that the counties with fewer newspaper subscribers were also more pro-Trump. (Correlation is not causation, etc.)

The authors appear to anticipate this complaint. (They say their results are significant even after controlling for college education and employment levels, but don’t mention if they did a similar check on ruralness.)

Many of the counties in Politico’s analysis of news subscription rates tracked with the urban-rural divide that was widely noted in the 2016 election, with higher-density population centers having higher subscription rates. But Trump appeared to struggle even in rural areas with a higher-than-average presence of mainstream media.

Yuma County, Arizona, for instance, had some of the features of the Trump strongholds — a heavily rural county on the Mexican border with sky-high unemployment (15.7 percent as of last November). But voters there tend to get their news from traditional sources: The county’s news subscription rate of more than 40 percent of households is in the top quarter of all counties in the country. Romney beat Obama there by 13 points, but Trump did 8 points worse, edging Clinton by 50 percent to 45 percent.

By contrast, there was Robeson County, North Carolina — the poorest in the state, with a similar economic profile to Yuma County — which has a subscription rate of only 9 percent, according to the Alliance for Audited Media. (The actual rate is somewhat higher, though, because one of the county’s papers, the six-times-a-week Robesonian, doesn’t report its circulation to the AAM.) Robeson County was a traditional Democratic stronghold, with Democrats having more than four times as many registered voters as the Republicans had. Obama swamped Romney there, 58 percent to 41 percent, in 2012. But Trump beat Clinton by more than 4 points — a swing of more than 20 points.

Giving two examples of outliers is not a particularly strong way to argue that we’re not just seeing rural/urban — but even these two outlier counties they mention are unusual choices. Both Yuma and Robeson counties are majority-minority counties that that also voted for Trump. Weird, right?

Yuma County is 60 percent Hispanic and smack dab on the U.S.–Mexico border. The fact that Trump had, well, plenty of things to say about Hispanics and the U.S.–Mexico border seems like a more likely factor in any anti-Trump swing than newspaper subscriptions.

(I’m also not sure I believe a “more than 40 percent of households” number for Yuma County. AAM lists exactly one daily newspaper in the county: the Yuma Sun, which reports a circulation of about 10,600. [It also has a Spanish-language sibling, Bajo El Sol, but that’s a free weekly.] The AAM audit of the nearest big-city paper, the Arizona Republic, says it doesn’t distribute in Yuma County at all; the Republic had exactly 9 subscribers in the county at last official count, according to AAM data. So that’s about 10,600 newspaper subscriptions in a county with 195,000 people and 64,000 households. Maybe I’m missing something — happy to be proven wrong here — but I don’t see how you get to “more than 40 percent” from there.)

Then take Robeson County in North Carolina. Only 29 percent of the county is non-Hispanic white; 38 percent are Native American, and another 25 percent are African-American. But note the huge exception thrown: one of the county’s papers, the six-times-a-week Robesonian, doesn’t report its circulation to the AAM.

Maybe the reason the county’s subscription totals are low is that, as far as I can tell, The Robesonian is the only daily newspaper in the county. If you ignore the only newspaper in town, then yes, newspaper subscription numbers do look low!

I don’t know how many subscribers The Robesonian has today, but as of 2009, it claimed a circulation of 14,000. (It also says it reaches 60,000 people a day between print and online.) Robeson County had about 44,000 households in the 2010 census. So even if that 14,000 number has dropped substantially, Robeson County is far from a news desert.

That Robeson fuzziness leads me to the final and most significant problem with Politico’s analysis. I could imagine being convinced I’m wrong about some of these other complaints — but this one lies at the root of the piece.

Politico relies on extremely incomplete and sloppy data. You may remember the Alliance for Audited Media by its previous name, the Audit Bureau of Circulations. Newspapers hire AAM to audit their circulation numbers — how many papers they publish and sell — so that advertisers can be more confident that the ads they buy will really reach as many eyeballs as the newspaper claims. All of Politico’s newspaper numbers come from AAM; if there’s a problem with AAM data, there’s a big problem with the analysis that relies on it.

Here’s another quote from the piece that illustrates that reliance on AAM data goes wrong:

Trump thrived, both against Clinton and compared to Romney in 2012, more strongly in the news-desert counties — the areas with very few subscriptions. For instance, rural Price County in northern Wisconsin had virtually no paid local-news circulation, according to AAM, and Trump walloped Clinton there by 25 points; the county had been a virtual tie between Obama and Romney in 2012.

Price County has virtually no paid local-news circulation? That would be breaking news to the Price County Review, the weekly newspaper there, which has in various forms existed since 1884. It’s part of a small chain of northwest Wisconsin papers, which is in turn owned by Adams Publishing Group, which owns a bunch of weeklies, mostly in the Upper Midwest.

I don’t know much about the Review, which charges $50 a year for a subscription. But it says that its print edition reaches 3,500 households. Price County only has about 6,500 households in total. Maybe some of those papers go out of county, or maybe their numbers are screwy somehow. But a paid print newspaper reaching something like half of a county’s households is no one’s definition of a news desert! It’s just that the Price County Review, like many hundreds of American newspapers, doesn’t bother to give AAM their circulation data anymore.

The story acknowledges that relying on AAM data isn’t perfect, but still argues that it’s pretty good:

Almost all daily newspapers report their subscription numbers, print and online, to AAM for verification in order to sell to advertisers. (Some of the smallest outlets do not, though, including weekly publications.)

That is just not true. “Almost all” large dailies report to AAM. But most small dailies don’t. And those small dailies are mostly in Trump territory.

It’s hard to pin down exactly how many daily newspapers there are in the United States, but the number is generally agreed to be somewhere around 1,300. (In 2014, the main newspaper trade association reported there were 1,331 American dailies; the number’s dropped a bit since then, but not by much. Editor & Publisher’s Databook currently lists 1,232 daily papers.) But AAM, according to its Media Intelligence Center database, only tracks circulation data for 668 of them. So AAM’s data is missing roughly half of American daily newspapers.

How about weekly papers, so important in smaller towns? Their trade group, the National Newspaper Association, says there are over 7,000 of those, about 44 percent of which offer paid subscriptions and aren’t just free giveaways. The NNA itself reports having 2,200 member papers.

So how many American weekly newspapers does AAM have data on? 428.

So who are these hundreds of missing daily newspapers and thousands of missing weekly newspapers located? It ain’t The Boston Globe. These are overwhelmingly small-town and rural papers. These are papers that don’t do much business with national advertisers, the main folks who like having audited circ numbers, and so they don’t think paying AAM to say their numbers are real is worthwhile. And what do we know about small towns and rural communities? They went for Trump by a wide margin.

That’s a huge gaping hole in this analysis. It’s not just Price County, Wisconsin and Roberson County, North Carolina that have badly wrong data. (Though, hey, that’s half of all the counties mentioned in the piece.) It’s hundreds and hundreds of counties. That omission makes subscription numbers in rural, pro-Trump counties look artificially low.

Again: I am absolutely willing to believe (and think there is at least pretty convincing evidence) that the decline of local newspapers has reduced engagement in the political process. I’d be very open to the idea that there is a statistical connection between the decline of local news sources and voting for Trump. (There’s plenty of evidence that Trump did extremely well among what political scientists call “low-information voters,” and there is a clear connection between low-information status and media consumption patterns.) The decline of local media is arguably the largest problem journalism faces today.

But this study doesn’t show what it’s claiming to show.