The phrase “user research” makes me think of Mad Men, with women sitting behind a one-way mirror trying on lipstick while men watch (and criticize their appearances).

Not surprisingly, this isn’t a good way to think about the concept. User research isn’t just for testing beauty products; it also applies to news sites, apps, and products. A one-way mirror isn’t required. User research is about seeing users as people with their own needs, interests, and agency. That makes it a very helpful tool for news organizations trying to figure out how to connect with the people reading their content.

There are a number of different approaches to user research, and “the best is the kind that the organization will do on a regular basis,” Karen McGrane, a user experience strategist who worked on the NYTimes.com redesign in 2006, told me. “The culture of an organization will lend itself to different types of research and testing, and different organizations will trust and value different approaches.”

I spoke with ProPublica and The New York Times — two very different news organizations that are pioneering user research in very different ways. They share a common goal: to shift away from thinking about the institution and toward thinking about customers.

“Most news organizations have never had to work in a truly user-centered way,” McGrane said. “For those looking to drive a new kind of digital culture, building more user research and testing into the process would probably be one of the fastest drivers of change.”

ProPublica: The guerrilla approach

If David Sleight, ProPublica’s design director, could tell other news organizations one thing about user testing, it’s that it doesn’t need to be complicated or expensive. In fact, you can just jump into it, as ProPublica did when Sleight was hired a little over a year ago, and think about it in a new way: putting your users in the newsroom with you.

“We’re learning about the audience and about the stuff we build, and what happens when we put it in their hands,” Sleight told me. “We don’t need to spend a million bucks. We are trying to do something in a very lightweight, inexpensive way that feels collaborative with our users.”

When ProPublica needs something tested, whether it’s an interactive feature like the surgeon scorecard or the site’s upcoming news app, the organization puts out a call on social media — “just asking who’s interested in taking a look at something we’re working on, whether it’s a layout for something or an interactive piece.” There’s also a form on ProPublica’s website that readers who are interested in testing products can fill out, and the email newsletter is another way to get in touch with people.

Want a sneak peek of upcoming @propublica news app? We're looking for user testers—email usertesting[at]propublica[dot]org

— ProPublica (@ProPublica) June 12, 2015

ProPublica then follows up with interested users to schedule a remote testing session. When the time arrives, the tester installs the free meeting and screen share plugin Join.me, and Sleight or assistant managing editor Scott Klein walks them through a product and asks questions.

“We work from a written script with some boilerplate questions that we customize,” Sleight said. “We do it on the fly. It’s much more jazz than it is classical music. The point is to capture impressions, see how people get through, and make them feel comfortable in discussion. The classic question we ask is, ‘How would you describe what you’re looking at to your friend who can’t see it right now?'”

Users are also asked to carry out tasks — looking up a surgeon, for example — and it quickly becomes evident when they’re stumbling over something or are unable to accomplish a task.

“Try as hard as possible to get out of the user’s way,” Sleight recommended. “People intuitively want to click, scroll, engage. For app projects, they want to touch and manipulate the thing that’s in front of them. The traditional way is to stop them, show them a lot of context, and then let them in,” but most testers just want to start clicking, so let them. “It’s about listening and being very comfortable with long, awkward pauses as people make their way through things,” Sleight said.

Heavy smartphone use has created a set of assumptions about how things will work. “Google and Apple have clearly shifted expectations around things like mapping,” Sleight said. “People immediately want to grab, click, manipulate, zoom in and out of maps,” and if they can’t, “they’ll tell you the map is broken.” Similarly, they expect to be able to click buttons, type text into entry fields, and get immediate feedback. “That’s a reasonable expectation at this point in time,” Sleight said.

ProPublica primarily goes outside the organization to find people to test its products, but Sleight said that when time is a factor, he’s not against going to someone inside the company: “I will look, literally, for the person who sits the furthest from this project and the people involved.”

For the most part, though, “we want to get outside the journalism bubble,” and that means also turning down outside journalists who answer ProPublica’s Twitter calls. “Especially in the Northeast and New York, we all pay hyper-attention to what other journalism shops are doing, but they are a very, very small set of our total audience,” Sleight said. “Our stuff has to be usable and understandable by a general population…Journalists are an extremely biased sample set.”

ProPublica records its Join.me sessions, and Sleight plans to start posting them for everyone in the newsroom to watch. “Once people [in the newsroom] start seeing this, it’s usually a total breath of fresh air,” Sleight said. “It’s a gateway drug to empathy with the audience.”

The New York Times: The formal approach

If ProPublica’s approach to user testing is “lightweight and inexpensive,” The New York Times’ is heavier, more organized, and more expensive.

The user research team is part of the Times’ Consumer Insight Group, which launched in 2008. Emily Goligoski, user experience research lead, and Maura Youngman, user experience research analyst, work with the news, product, marketing, and advertising groups. (Their presentation at last month inspired this article.)

After a team submits a research request, Goligoski and Youngman work with them to create a plan. They may incorporate analytics to get an overview of what users are doing and in what numbers; from there, more qualitative research helps them dig into users’ motivations and behaviors.

“We have a very deliberate conversation about the role of the research and what groups you’re reaching out to,” Youngman said. “If the answer is ‘everyone,’ that answer is wrong.” Part of Youngman and Goligoski’s role is to introduce other teams to the various research methods that are available to them — surveys, in-person interviews, testing — to determine the best one for a given project.

The Times uses a number of different methods to recruit users who are interested in testing products. “If we’re looking for users who are avid users of a Times product or core site, we’ll put up a survey asking a bunch of different questions, look at that list, and recruit from there,” calling respondents and scheduling times for them to come in to the office or talk on the phone, Youngman said.

The Times also does intercept testing on its site with software called Ethnio, which pops up and ask people to take a survey of 5 to 7 questions, then collects their contact information for same-day followup. “If you want to diversify your pool on the fly, live intercept testing gives you that flexibility — it’s not people who have been booked to come into the office for two weeks,” Goligoski said.

When the Times needs to talk to people in a very specific demographic, it hires an outside recruiting firm to find them. It also sometimes sets up user research stations in secondary markets, like Chicago and San Francisco, or in international markets, like Mexico and Brazil.

“The broad cumulative effect of user testing has been a stronger sense of empathy across the company,” Mark Silver, lead UX strategist, said, echoing ProPublica’s Sleight.”There’s an immediate benefit there, and also a subtler one, which is that people [at the Times] think more in terms of users now, and less in terms of institution.”

… But I had user research scheduled. :( https://t.co/4em8GUEdDi

— Maura Youngman (@mauramaura) June 12, 2015

In one recent study, the Times asked users to monitor their news consumption habits over the course of a week, using an app called Revelation. During that period, two major stories developed — the Germanwings crash and the Kenyan university shootings — and there were interesting patterns in the ways that readers consumed these stories.

“Early followers,” Goligoski said, “started following the story immediately and really couldn’t get enough.” Once they had the basic facts, they read tangential stories on topics like mental health and employment or aviation safety. “Latecomers,” meanwhile, entered a story several days in. Perhaps they previously hadn’t cared about it, or it hadn’t seemed relevant, but then “it ended up taking up so much space in their social feeds that they felt the need to know.”

“We saw Wikipedia serving the latecomers’ needs really well, because it’s written in chronological format,” Goligoski said. “Participants felt it’s vetted, and it shows up prominently and consistently in their search results. That was something we would not necessarily have found” without doing a diary study.

In another study, last fall, the Times looked at a specific subset of millennials: 18- to 34-year-olds who were heavy news consumers, followed different types of media, and subscribed to print magazines. The user experience team conducted 60- to 90-minute interviews with a few dozen of them, and visited 9 in their homes.

“One woman read NYT Now on her daily commute from Brooklyn to Manhattan, and together with a videographer we went along with her on one of her morning commutes,” said Goligoski. “We wanted to see the moments of joy and delight, and the moments of distraction.”

When the researchers visited interviewees who worked from home, they saw other interesting patterns: “People will tell you the start checking emails when they get to work, but what we saw was that there’s a lot of news reading from bed, or from the couch, or over cereal. The news consumption starts the moment the alarm goes off, and doesn’t stop until 20 hours later.”

From the millennials research, the team ended up creating a 15-minute video that it has shared across the organization.

The Times clearly has significant resources that smaller news organizations, including ProPublica, don’t. But there is “low-hanging fruit,” Youngman said. “We’ve done conversation facilitation in-house. That’s a few pizzas and sodas, something most news organizations can tackle.” She also mentioned UserTesting.com, a crowdsourced site that gets participants to look at prototypes or applications and quickly provide feedback.

Goligosk’s best advice for other organizations is to “get really specific about what you want to learn. Go through the exercise of identifying areas where UX research is particularly well suited to tackle your questions. It sounds really simplistic, but, especially when you don’t have a lot of resources, [you need to] really think about the specific questions, as opposed to trying to tackle every team’s product.”

And journalists should keep in mind that user research is actually a practice that fits with their natural skills. That might make its practice in the newsroom seem less foreign.

“Good interviewing, in so many ways, mimics quality reporting and information gathering,” Goligoski said. “Being an active listener, thinking toward the narrative you want to share back to your colleagues, really being driven by facts. You’re both trying to get to the ‘why.'”