The demonstrations sparked in response to the murder of George Floyd are drawing participants with a range of political and ideological motives, including some allegedly less interested in protesting police brutality than promoting the aims of extremist groups. The Department of Homeland Security, for example, has specifically warned about domestic terrorists exploiting the protests to spark violence and other officials have warned of disingenuous actors, though they disagree on who to blame.

What might you want to know while covering a demonstration? Well, for starters, that wearing a Hawaiian shirt and igloo symbol could indicate a thirst for civil war.

A group of researchers and journalists at Columbia University has been thinking about the problem since photojournalist and Columbia professor Nina Berman saw a woman at the 2018 Unite the Right 2 rally in Washington, D.C. with distinctive tattoos, including one that read “1488,” a white supremacist meme. If she’d been able to recognize the reference, Berman lamented, she would have asked the woman about it directly.

Enter: VizPol, an app that aims to help journalists covering demonstrations identify emblems in real time. Built by researchers at Columbia University’s journalism and engineering schools and launched as an invite-only beta this week, VizPol can currently recognize 52 symbols. Many are associated with right-wing and white supremacist organizations, but the app also includes insignia used by libertarian, anti-fascist, hactivist, and other groups considered political but not extremist. (There’s not yet an entry for “Boogaloo Bois,” the militia group associated with the Hawaiian shirts and igloo imagery.)

While the “1488” mark was relatively easy for Berman to look up — albeit after the protest — other marks are harder to Google.

“There’s a group called Atomwaffen Division that has an obscure symbol that looks a bit like a nuclear hazard sign,” said Ishaan Jhaveri, research fellow at the Tow Center for Digital Journalism at Columbia. “How do you search for that if you don’t know how to describe it?”

Jhaveri, an occasional birdwatcher, said the app was partially modeled on Merlin Bird ID, which identifies species of birds. In VizPol, journalists upload images and frame the symbol they want to identify within a red box. To create the robust reverse image search, the researchers reached out to more than two dozen photojournalists for images from demonstrations.

Berman and Jhaveri both emphasized that journalists should not treat the app’s identifications as confirming that an individual belongs to a particular political and/or extremist group. Some symbols have been appropriated from other political or cultural contexts, for example, or the individual may be unaware of the associated group’s beliefs. (A football player drafted by the New England Patriots recently claimed he thought his tattoo of the logo for the Three Percenters, a far-right militia group, stood for supporting the military.)

“It’s very important to state — and for users of the app to know — that the app can only go so far. You give a symbol and it gives you some context about that symbol and some groups that have been known to use that symbol,” Jhaveri said. “But that doesn’t necessarily mean that this person adheres to the group’s ideology.”

As one example of the changing nature and meaning of symbols, Jhaveri noted that pro-democracy demonstrators in Hong Kong have adopted Pepe the Frog, a cartoon that regularly appears in connection with pro-Trump and anti-Semitic material stateside.

“It serves more as a tip sheet introduction to ideologies of various political groups in order to provide context so journalists can ask better questions, or take more informed pictures, and caption them with more specificity,” Berman added in an email. “When used, it also encourages the journalist to start looking for symbols which can lead to a more investigative instead of reactive approach.”

There were a number of tricky technical problems to overcome. Today’s political symbols often originate online but wind up on flags, hats, patches, armbands, T-shirts, skin (as tattoos), and more. The same symbol might be professionally replicated for members or hand-drawn by someone who subscribes to the ideology from afar. The Columbia team had to train the app’s algorithm to recognize symbols in a variety of contexts and renderings.

“For example, the Three Percenters have a flag with their emblem but they also have patches that their members wear and the texture of the flag and the texture of the patch is quite different,” Jhaveri said. “A human can immediately tell that it’s the same symbol on two different textures, but for a computer to be able to treat those two images as the same symbol is much more difficult because the computer is more responsive to those changes.”

The app’s characterizations of symbols and associated groups draws on the extensive research on extremist and other political groups done by the Anti-Defamation League and the Southern Poverty Law Center. Since the app is meant to be used by journalists in the field, the descriptions are short — fewer than 200 words — and hardly comprehensive.

It’s a start, though, and one that the Columbia team thinks is badly needed. Berman said their outreach showed that editors, even those on political beats, were unable to recognize “a good majority” of the symbols the app lists.