Nieman Foundation at Harvard
Editor’s note: The email newsletter consultancy The Edge Group is talking to a number of the leading thinkers in the space to see what lessons can be gained from their approach to email as an editorial channel. I thought this piece, looking at the newsletter-driven publisher Axios, has some interesting detail worth sharing with Nieman Lab readers.
Would you consider your relationship with a reader that opens 100 percent of your emails the same as someone who opens 0 percent? Obviously not. Then why do we still think in the aggregate when looking at email list metrics? Most publishers remain mired at the campaign level, ending their newsletter analytics exploration with list-level open and click rates. Are there ways to go deeper?
When we — The Edge Group, a consulting firm focused on email newsletters — sat down to speak with Rameez Tase, vice president of growth at Axios, we were treated to a principle that beautifully encapsulates both the problem and the solution.
Let’s call it Rameez’s Law: You don’t get a 50 percent open rate by having a 100 percent opener and a 0 percent opener. You get two distinct cohorts that you act upon in different ways.
Some of your audience is engaged, and some of it isn’t. So why do we treat them all the same? We shouldn’t measure success as an aggregate — we should instead try to understand if the right people are highly engaged.
This is the essence of Rameez’s Law, and there’s beauty in its simplicity. Think about your audience at a deeper level than averages and aggregates. By moving beyond campaign-level stats, down to cohorts and user-level data, you start to grow an audience while building lasting relationships, maintaining trust, and paving a solid, sustainable future for the company.
In this article, we lay out what Axios is doing to achieve this. We’ll cover a number of frameworks to help elucidate why this approach works: Goodhart’s Law, Andrew Chen’s Law of Shitty Clickthroughs, a DTC brand mentality, and much more.
So how does Axios do it? How does a media company leverage user-level data to rapidly grow, while still maintaining trust and loyalty? Before we dive further, let’s take a look at who Axios is.
We’ll borrow our description of the company and its founders from a post we published last year:
Jim VandeHei and his two-time co-Founder, Mike Allen, perfected the art of digestible, must-read daily emails with Politico’s Playbook. Axios takes the elements that made Playbook a success and instills them across an entire company. These elements are summed up in the term “Smart Brevity,” which implies no wasted words and no wasted space.
Tase brings experience from both the editorial and the financial worlds. He was an analyst at Deutsche Bank and the Blackstone Group before transitioning to a three-year stint as vice president of audience and analytics at Mic. That mix of business and journalism is omnipresent at Axios offices — skilled analysts know something about the business side, and people skilled on the business side know a bit about analytics. This creates a virtuous circle where everybody makes each other smarter.
A big part of Tase’s job is to figure out the levers, both paid and organic, that produce engaged newsletter subscribers. Web traffic is one funnel — website readers who haven’t yet subscribed to an Axios email. Out of the 10 million unique monthly visitors they get on their site, Axios has 1.5 million non-unique subscriptions across all 19 of their newsletters.
The other major funnel is social, where Tase can find people who seem like the ideal Axios reader but haven’t yet encountered the brand. As social has plenty of tools for isolating preferred demographics — lookalikes of Axios’ engaged users — this is an ideal channel for finding loyal readers.
Thinking about those two as very distinct funnels is an example of a segmentation mindset. One potential subscriber knows your brand and goes directly to you. The other you’re going out and locating yourself. The way you work to convert these two cohorts will naturally be distinct.
CPL, CPC, CPI — cost per lead, click, and install, respectively: in Axios’ eyes, these common marketing measures fail to capture the true cost and ROI of acquiring an email subscriber. Many media companies would agree. If they’re running social ads to get newsletter readers, they may measure CPEA — cost per email acquired. It’s a step in the right direction, but Axios goes one step deeper.
They think about CPEEA: cost per engaged email acquired. That number takes into account unengaged user churn. An email address isn’t just a line item in Axios’ book. To understand the true efficiency of your marketing spend, you should understand whether or not you’re investing in signing up engaged readers. “Getting an email address is the lead — converting that email to an engaged subscriber is the goal,” Tase says. Once you have them, you need to keep them.
When Axios asks for something, their community responds: User surveys get a 10 percent response rate. Their vanity metrics are also way above industry standards (they average an open rate of 45 percent). Engagement is, by all measures, through the roof.
But Tase has to add a caveat: “One of the things with email analytics — they always tell you to do more.”
Or to put it another way, email analytics will never tell you to do less. You’re not going to get metrics indicating you need to ease up on ads when click rates are high, or that you should push hungry readers less content if they’re consuming everything you send. “You never know when you have eroded trust by that next metric,” Tase says.
He notes that it takes intuition and an objective outlook to balance engagement and avoid losing trust. Measuring trust is difficult, and its level often only becomes clear after a breach has taken place. This is where we need to look at Goodhart’s Law.
Charles Goodhart is the economist whose famous adage holds that when a measure becomes a target, it ceases to be a good measure. Wikipedia gives the example of a car dealership where the success metric is the number of cars sold. This will lead to employees optimizing for the number of sales at the expense of profits. People will alter their behavior, potentially for the worse, in order to maximize whatever is measured.
A typical Axios newsletter has six to ten “cards,” each summarizing a noteworthy news item. Readers can skim until they hit stories they enjoy, read the whole thing top-to-bottom, or any other path they choose. What’s important is that they’re engaged.
What’s less important is a metric like time spent reading — a bit of a holy grail in the newsletter world. As Tase notes, “the easiest way to keep someone reading longer is to make the content…longer.” It’s tongue-in-cheek, but it captures the essence of Goodhart’s Law.
That doesn’t mean Axios doesn’t pay attention to more traditional numbers. List size and open rates are metrics advertisers care about, after all. But when Tase thinks about his role, he sees cohorting as a way to make human decisions more manageable.
What in the world does cohorting mean? To answer, we’ll first have to take a look at another law.
Numbers are great at Axios. They have some of the best stats in the industry. But Tase is well aware of the dangers of a false sense of comfort. Newsletter consumption is linear — it takes 5× longer to read five newsletters vs. one, and every reader has limited time. How can they push the business forward knowing this limitation?
At some point, Andrew Chen’s Law of Shitty Clickthroughs starts setting in: Over time, all marketing strategies result in shitty clickthrough rates. The novelty wears off; the market gets more crowded; your new customers aren’t as good as your old ones were.
Axios wants to build real brand equity. They know this age of plenty in email won’t last forever. Tase’s team is doing everything it can to futureproof the business. Part of that means obsessing over the correct measurement of lifetime value.
Acquiring engaged readers reaps continual benefits that go beyond their individual usage. For the most part, “an 80 percent open rate user begets another 80 percent open rate user” when they share the newsletter. For the sake of consistency, let’s call that Rameez’s Second Law. (Surprise! There are two.) “We really want to focus on who is this person, what value can we bring to them, what motivation can we provide them to share this product with their personal or professional networks,” says Tase.
With user-level data, he and his team are able to hone in on distinct groups with social retargeting, surveys, ambassador reward plans for signups, and other efforts to bring more highly engaged readers to Axios.
Cohorting is a way to make human decisions more manageable. Can you imagine blasting these sorts of campaigns to your entire list? Both the zero-percenters and the hundred-percenters? It would be absurd. User-level data and segmentation are there to bring the right people into the community and keep the business sustainable. So is a quality product readers can trust.
Axios caters to a premium group of loyal readers (decision makers) in an intimate setting (email). That earns the company expensive sponsorships. That allows them to afford slightly more sophisticated marketing efforts on the cost-of-acquisition side. That grows their list. That grows their business.
Some in the journalism profession might whip out the garlic and wooden stakes at this point from Tase, but hear him out: “We need to think much more like a direct-to-consumer brand.”
These companies, DTCs, “are very sophisticated because, at the end of the day, they need to sell a product. Casper or Brooklinen can’t really fake it. If they don’t sell a mattress, then they wasted that money acquiring the email. Media companies have been able to fake it when your customer and reader aren’t necessarily the same.”
A traffic-based or display-advertising-based media business isn’t held accountable for the same sorts of things a DTC is. They can nudge your numbers higher for a while with tricks like arbitraging paid traffic against display ads. But that sort of a strategy doesn’t translate to an engaged community that will stick around.
That’s both an opportunity and a challenge for Axios: The business’s future depends on a readership that really reads. And people read Axios. The challenge is that growing a portfolio of newsletters could decrease the engagement each one gets per edition, because there are no economies of scale in newsletters. Remember: Reading five newsletters takes five times as long. People only have so much time.
So what does Axios do? They start crunching numbers — specifically lifetime value.
LTV can be a dirty topic for a journalistic publication, and it’s a difficult one to measure in an advertising-based business, but it’s now a mandatory level of strategic thinking. Growing an engaged community is the smartest thing Axios can do now.
Besides qualitative discipline — maintaining unique personalities across newsletters, creating content that doesn’t waste readers’ time, providing a diverse set of complementary newsletters to choose from — Axios is able to grow an engaged community by leveraging user-level data. They’re not segmenting users for a marketing automation algorithm, but instead to make data more accessible to humans. Remember, they’re cohorting to make human decisions more manageable.
They’re dealing with a community in the millions, yet even on the growth and analytics side — which in most organizations is a relatively detached, numbers-only group — they’re putting the human element first and foremost. Axios manages to avoid the vicious circle that most organizations face when they don’t heed Goodhart’s Law, blindly follow the metrics, and disintermediate themselves from their audience, losing trust in the process. Axios instead establishes a virtuous circle of organizational transparency, inter-team communication, a multidisciplinary workforce, and organizational DNA that is rooted in every team member.
Combine “the person with business context with the person with data infrastructure context, and they both make each other smarter,” Tase says. “Also, anyone can get the data they need.”
Maintaining that virtuous circle depends on everyone having the tools they need. But in the newsletter world, “all ESPs have a host of chokepoints, because the technology of email is pretty archaic,” Tase says.
When newsletters are your bread and butter, there aren’t many tools on the market truly geared towards your goals. Everyone we’ve spoken to for what will be a series of articles on these issues — from Tedium’s lone warrior Ernie Smith to SmartBrief, a two-decade newsletter veteran — has relied on custom-built tools that help them “newsletter” better.
Axios is no exception. They have a custom-built CMS called Eden. This hyper-personalized tool allows for smooth newsletter publishing, with Axios standby sections like “Go deeper” available in the editor as a button and the ability to publish to both the site and a newsletter in one fell swoop.
Axios takes readability very seriously; the CMS is built to force writers to write mobile-first. (Apparently, some writers write entire 1,500-plus-word newsletters on their phones. You heard it here first, folks.)
Axios still uses third-party tools. Sailthru’s API is used to send emails via Eden, while also handling the data collection and aggregation. Axios uses Data Exporter for event- and user-level data. (“We [also] use BigQuery and Tableau. This allows us to move from campaign-level data, to breaking down building blocks and reassembling them into user-level data.”)
Axios is a lot of old-school notions of journalism wrapped up in business savvy and the latest publishing tech. From early-a.m. news cycles to custom-built tools; from user-level data analysis to a DTC brand’s mentality.
But for all the robust infrastructure, industry know-how, and quality writing, it all leads back to that simple adage: You don’t get a 50 percent open rate by having a 100 percent opener and a 0 percent opener. You get two distinct cohorts that you act upon in different ways.
Axios is successful because it’s able to see the obvious in something so obvious it goes over most media company’s heads: You don’t treat your engaged users and unengaged users the same. Once you’ve internalized this, you can start thinking about your email list in new ways. Rather than just checking a box for hitting a campaign-level open rate, you can start thinking about how to build deeper relationships with your engaged users. You can work to scale your audience while maintaining the trust required to build long-term relationships and realize a higher lifetime value. It’s a critical mindset shift when you’re building a valuable, sustainable media business.
Aleks Smechov is senior data analyst at The Edge Group, a consulting firm focused on email newsletters.
