I’ve been writing this week about The Associated Press’ plans to rethink what it means to be a wire service on the Internet. Much of the reporting began with a document entitled, “Protect, Point, Pay — An Associated Press Plan for Reclaiming News Content Online,” which was distributed to AP executives, board members, and some members late last month. Though I have some more to say tomorrow, this seems like a good time to release the seven-page document in full. Here are my three posts so far:

— Why The Associated Press plans to hold some web content off the wire
— How The Associated Press will try to rival Wikipedia in search results
— What The Associated Press’ tracking beacon is — and what it isn’t

I spent an hour on the phone with Srinandan Kasi, the AP’s general counsel, on Tuesday night, so if you have questions about it, feel free to ask, and I might have an answer. You can download the document, view it in Scribd below, or wade through the full text after the jump.

UPDATE, August 14, 3:35 p.m. Andy Baio, responding to a commenter’s request, kindly ran the document through an OCR scan. Thanks, Andy! Two images from the original are not included, of course, but here’s the full text:

Protect, Point, Pay – An Associated Press Plan for Reclaiming News

Content Online: Part I

The evidence is everywhere: original news content is being scraped, syndicated and monetized without fair compensation to those who produce report and verify it. AP’s legal division continues to document rampant unauthorized use of AP content on literally tens of thousands of Web sites. The problem is quickly spreading to mobile, where new applications are cropping up daily that do little more than repackage the efforts of AP and others, siphoning off consumers and revenue from those whose content is being exploited.

If there is good news, it’s that the issue is now getting broader attention. A group of newspaper publishers, coordinated by the NAA, has organized to review new media challenges and opportunities. An influential federal appellate judge, Richard A. Posner, has advocated for changes to copyright laws to protect content originators. The American Press Institute (API) has recommended models to migrate online content from free to paid. And an array of entrepreneurs and organizations has cropped up to suggest various technology and business solutions.

In the past three months, AP has solidified its plan to address the urgent need to regain control of news content online, beginning with AP news. Dubbed AP3P by Tom Curley for “AP Protect, Point and Pay,” this three-pronged initiative provides a blueprint for AP to move forward in the digital world and offers a model for restoring the worth of authoritative journalism online.

It is difficult to overstate the importance of taking action at this moment. With its traditional media customers under unprecedented financial pressures, AP simply can’t continue to provide the same quality of global news coverage under the current rules, where secondhand news gets most of the eyeballs. Emboldened by the uncertain state of the law around content use online, third parties are moving quickly to fortify their own positions. AP has both business and legal imperatives to assert its intellectual property rights, make affirmative efforts to protect them and create a structured way to enforce them. Taken as a whole, the AP3P initiative will give AP a firm footing on which the cooperative and its members can build a successful platform for innovation. Its success, however, depends on cooperation of its membership and careful execution of each part of the plan.

Described below are the foundational steps of AP3P that we believe are necessary for AP to gain control of its content. We’ve also outlined ways in which interested members (a “content coalition”) could take advantage of the protections AP is building for itself to protect their own content. In Part 2, we will address how AP and its membership could leverage the work done to gain control of our intellectual property to build content-rich products that create revenue streams for the news industry.

AP Protect

Core to AP’s plan is identifying and protecting its news assets. AP Protect takes advantage of the underpinnings of AP’s strategy that have been in place since 2006 when it created “eAP” to database, catalog and format its content for wide-ranging digital uses. The eAP database established a central warehouse for all AP’s content, across media type. Metadata coding followed, to organize the content by category, famous names and other attributes. This past year, AP developers created a standardized news “microformat” to ensure that every piece of content can carry key pieces of metadata, plus AP’s terms of use and a “digital beacon” to track usage.

The methodical build-out of eAP positioned AP over the past 18 months to begin offering a set of services to members through the Digital Cooperative program. Those services started with metadata mark-up of member content and now include the AP Mobile initiative, as well as content exchanges among members via AP Exchange. Work on the news microformat, known internally as “hNews,” as well as development of the beacon has now advanced to the point that it can serve as the basis for another valuable service, AP News Registry.

In a breakthrough announcement earlier this month, AP’s microformat was formally endorsed by The Media Standards Trust, a global research and development group devoted to fostering higher standards in news. The trust is calling on news organizations worldwide to adopt more consistent news formats for online content. Broad adoption of this simple open-source standard way of describing news and content rights could quickly change the balance of power for content owners online by providing a uniform way for search engines and others to surface it.

AP News Registry

AP News Registry is a way to identify, record and track every piece of content AP makes available to its members and other paying customers. What makes it different from other similar efforts is that it is being designed at the outset with a rights framework that will provide an enforceable way for AP to grant and monitor specific rights in its content.

When AP distributes a news item, that content will be wrapped in a container that will include rights information and a tracking beacon that will send reports back to the core database each time the item is clicked on by an end user. The beacon will identify each piece of content, the IP address of the content viewer, the referring Web server and the time of use. The content will also be wrapped in a simple piece of code that will travel with the content wherever it is posted on the Web and that spells out, in both human and machine-readable forms, what may and may not be done with the content. This is the microformat that is described above.

Because the News Registry’s active tracking beacon would not be effective if the beacon were removed, the Registry also has a backup enforcement system. Based on current Web behavior, it is safe to assume that some users will intentionally or inadvertently remove the beacon. A “passive” tracking service will crawl the Web searching for AP content and identify the publishing Web page, an image of usage and the time of discovery. Matches will be queried against the active tracking database, and unauthorized uses will be pursued.

AP News Registry will leverage AP’s in-house expertise in building and maintaining content databases, tagging content with metadata and distributing content as well as reuse some components that make up the eAP infrastructure. The registry will also require capabilities not currently available in the eAP platform. Those components either will be developed in-house or acquired through strategic partnering.

Opening the Registry to Other Content Owners

AP News Registry is a necessary addition to the AP infrastructure. However, the potential benefits of a news registry to the U.S. newspaper industry and other news content providers is equally compelling, and widespread adoption of the standards that would be part of AP News Registry would be beneficial to those trying to replace rapidly declining revenue from traditional media with online revenue.

In particular, AP News Registry could provide members with:

1) Better business metrics. A rich database of content usage and payment data would give publishers a more complete view of how their content travels and is consumed.

2) More protection. An automated permissioning framework would allow publishers to gain control over how their content is displayed and distributed online.

3) Flexible payment models. The rights framework AP is creating will support all transaction models that get adopted by the market, from micropayments, to paywalls to advertising.

4) Leverage with partners. The widespread adoption of common standards would improve the industry’s ability to negotiate with partners, including search engines and content aggregators.

AP is working toward launching AP News Registry in beta test for tracking and reporting of AP content by Nov. 15, 2009. In Phase 2, it would be expanded to include content from members of the cooperative, beginning in Ql 2010. A production-quality release for AP and member content is targeted for July 1, 2010, accompanied by on-boarding a large number of content owners.

AP Point and Pay

Implementation of AP Protect through the AP News Registry is fundamental to the next two P’s in the AP3P plan: Point and Pay. These are simply a shorthand means to describe a whole new approach AP is taking to online product development to address the revolution in how people consume information in the digital era.

The purpose of Point is to leverage AP news content and information management tools to harness online traffic in ways that reduce misappropriation, expand audience and deepen engagement. Pay seeks to maximize revenue by aligning the commercial terms under which AP’s products and services are available on various platforms. It also includes the ability to monetize premium content in new ways, such as highly targeted advertising support and pay models.

AP management also has been reviewing the offerings of several entrepreneurs in this area and plans to continue working with some.

Clearly, search and social networking have transformed the ways a new generation gets its news. In Part 2 of this report, we describe patterns of news consumption around the death of Michael Jackson that demonstrate just how far we have come from the bookmark-your-favorite-Web site model of just a few years ago. Each day, more and more news is consumed somewhere other than on traditional destination Web sites.

The AP3P plan involves segmenting AP’s online products to broaden redistribution of what we call “utility” content, i.e, the type and amount of news that is quickly and easily available from other sources, to limit or prevent redistribution of the kinds of information AP provides uniquely to ensure that hypersyndication does not drive down its value, and to create a “news guide” in the form of landing pages to serve as a focal point for discovery of authoritative sources of news.

Over the last few months, we have been conducting a careful inventory of AP’s online content, how it is used and which attributes, features and content types are easily replaceable by other sources and which are uniquely created by the cooperative. We are now in the process of defining a new online product set segmented for different customers and licensed uses.

Taken as a whole, we believe AP3P has great potential to transform the way AP and its members do business online and ultimately unlock enormous new revenue potential for the content industry as a whole. Making this transformation, however, will not be simple, and comes at a time when the industry and the economy as a whole is under huge financial stress.

We believe AP3P offers the opportunity to change the conversation with the portals and drive traffic Рand new revenues — to original journalism. In the next section, we outline some of the ways this might work.

Part 2: Protect, Point, Pay -The Revenue Opportunity

In the past few weeks, amid all the industry discussion of new digital business models, a new milestone was reached in Internet news consumption.

Michael Jackson died suddenly on June 25, and within 30 minutes, the news absorbed 25 percent of all Web traffic. Online news sites logged an astounding 4.2 million visitors a minute, according to the delivery network Akamai.

Two of the biggest beneficiaries of that traffic bonanza were Twitter and Wikipedia, a couple of digital natives that would have been viewed as very unlikely news competitors even a few months ago. Indeed, a new pattern of consumption was validated in the confusing minutes that followed the first reports of Jacko’s death: Users shared; they searched and they clicked on Wikipedia.

In the course of only a few hours on the first day of the story, the Michael Jackson page on Wikipedia received 1.8 million visits. By Friday, the total reached 5 million visits.

For those with long Internet memories, the new routine of Twitter-to-Google-to-Wikipedia contrasts sharply with the behavior of users in August of 1997, when millions loaded and reloaded bookmarked news sites to get updates on the death of Princess Diana, another celebrity icon of similar magnitude.

The traffic pattern also represents an important variation in the established drill since Sept. 11, 2001, when users began using Google and other search engines as a shortcut to news snippets.

Now the news may be shared before it is even searched. And, what’s more, in cases where famous people, places and things are involved, you will undoubtedly find Wikipedia in the mix, with its battery of standing pages that are updated continuously and built to send people where they are looking to go.

The Wikipedia page on Michael Jackson is not very pretty to look at, but it has more blue hyperlinks than black type. Forget the “wiki” method of community updating, the key to Wikipedia’s success is that its pages are designed to catch traffic, provide key information and then send users on their way to deeper engagement on the subjects they’re interested in. In the four weeks ending July 4, the key search terms delivering visitors to Wikipedia were mostly famous names – Billy Mays, Farrah Fawcett, Megan Fox, Ed McMahon, David Carradine, John Dillinger, Steve McNair, Karl Maiden – all in the news for a variety of reasons, along with Jackson.

(Note: Google News was the single largest beneficiary of traffic for “Michael Jackson” searches, attracting 7.1 percent of the clicks in the four-week period measured. Wikipedia was a close second, receiving 6.8 percent. YouTube was third with 6 percent. CNN.com, the only traditional media outlet in the top 10, was tenth at 1.5percent.)

If the rise of Google taught people to search rather than surf Tor news, the phenomenon now seems to lead more often than not to a comprehensive, perpetually expanding resource called Wikipedia. Year over year, Wikipedia has gained impressively in every aspect of audience measurement, from total visitors (nearly 60 million) to minutes per month (19).

The lesson is as much a cultural case study as it is a competitive alert. With the help of social nets like Twitter and resources like Wikipedia, users are gaining even more control over their news consumption than Google originally provided through search and, later, Google News.

Moreover, if past milestones like Diana and 9/11 are any indication of how situations can solidify habits, the choices being made right now, amid a parade of famous-name news stories, could become routine. To that point, there was evidence in the early minutes of the Jackson story that visitors to Wikipedia were struggling to confirm the death. Editors at Wikipedia openly parried with people trying to make updates on the Jackson page to call the death. “Once again, he is not dead, just stop,” wrote one exasperated editor in a posting to the page.

But clearly, based on the numbers, that debate – and search for truth – didn’t seem to discourage the onslaught of traffic to Wikipedia, which was functioning, at least for some large portion of the Internet audience, as an authoritative source working to verify an important news development.

What It Means for AP and the Larger Industry

Traditional news providers are not necessarily built to exploit this activity. Destination Web sites, open or closed, are the principal resources that traditional publishers bring to this competition. More and more, as recent events demonstrate, those tools don’t completely measure up to the job. In the Jackson episode, publishers lagged the field distantly. The Los Angeles Times, at the heart of the story, managed to capture 0.79 percent of the “Michael Jackson” search traffic. It was the lone American newspaper site in the top 20. These numbers suggest that more publisher content was likely consumed through search engines and aggregators than on site, which is the crux of the issue now facing publishers.

To be sure, sites have optimized their news pages for search, and some publishers are now pushing out news and links via Twitter. AP Mobile, the first news organization to create a ‘push” function on Version 3.0 of the iPhone, sent a news alert on MJ’s death that resulted in a 350 percent spike in AP’s mobile traffic that day. But the war is being fought on several fronts at once, presenting a range of responses to consider.

For one thing, the Wikipedia model of standing, authoritative pages could be challenged. Beyond that, the deep resources of publishers could be aggregated for easier access; the mobile channel could be opened more widely for search and social consumption of news; and new applications for delivering personalized content directly to users could be plugged into publishers’ sites. These are just a few of the moves that could tap into the new demand that’s emerging from increasingly empowered news consumers.

These opportunities align directly with AP3P strategy described earlier. The three P’s truly become capital P’s if they can harness the scope and scale of the Digital Cooperative’s content and user base. Currently, more than 1,200 AP members are taking part in the Digital Coop’s content enrichment program, and about 1,000 are signed up to take part in the first commercial application of it, AP Mobile.

The microformat standard that is part of AP Protect could give the industry leverage in broader discussions with search engines and integration with social networking tools as they are developed. AP News Registry could provide detailed usage data across the industry. AP’s concept of a news guide could become the first step in a wider platform for the news industry to create richer products and foster deeper audience engagement.

A large and diverse content inventory also should provide new opportunities for search, contextual and behavioral advertising. Virtually all of the new pay models that have been proposed to publishers in the past several weeks are based on those assumptions. The network effect of plugging such a sprawling inventory into search engines, portals, social nets, mobile channels and pay applications could be substantial, just based on the lessons now being learned from the Wikipedia phenomenon.

A recent whitepaper from the American Press Institute has independently arrived at similar conclusions. The Newmedia Economic Action Plan recommends a “consumer-centric” model for publishers that moves beyond traditional Web sites. The report advocates developing new applications for users, charging for some content and using IP protection to capture more value from syndication. It further urges collective action – including “joint supplier negotiation” ~ to extract fair compensation from digital distribution channels, such as search engines and aggregators.

As the Web morphs once more and the culture shifts even more decidedly in the digital direction, these are no longer just theoretical opportunities. The scope of coverage that can be exploited from AP and its members constitutes a clear competitive advantage in an increasingly fragmented news marketplace. As the smartphone applications already attest, the combined content base covers all geographic bases and multiple news categories. Based on consumption patterns and competitive trends, they represent real choices to be made, and soon.

Next Steps

We’ve focused in this document on the immediate steps AP proposes to protect its own content, and set forth the larger opportunity for a coordinated effort by a coalition of members of the newspaper industry, based on the same principles.

Clearly, the organization and operation of any coalition must reflect pro-competitive principles and must not violate applicable antitrust and competition laws.

Any coalition must begin with shared values. The obvious overall aim of our effort is to support and promote authoritative journalism while protecting original content from unlicensed use. While the Internet has opened up exciting new opportunities for others to provide firsthand accounts and to comment on and share news, the critical role professional journalists play in newsgathering, sourcing, fact-checking and curating has been undervalued. Even as much of the news online still originates with newspapers, consumers often end up reading second-hand and, often, inaccurate versions. The coalition would align around developing business models that support quality journalism and encourage consumer interaction with news in a rights-protected environment.

The focus, then of the content coalition, would become:

* Participation in AP Protect and AP News Registry;
* Standards and principles for dealing with content aggregators;
* Shared content monetization systems; and
* Combined product development and syndication opportunities.