MediaCloud: Charting News Trends Online

During this unfolding drama of the death of the media industry as we know it, we face the constant choice of whether to essentially look backwards, and chronicle the collapse of one sector after another; or look forward and document the birth of a new media system right before our eyes.

Now that my colleagues in traditional media are taking up the first challenge, as witnessed by today’s excellent roundup in The New York Times by Richard Pérez-Peña headlined “As Cities Go From Two Papers to One, Talk of Zero,” I am going to try and focus on the less well-charted territory of our common media future.

For help with that, today let’s turn to Harvard’s Berkman Center for the Study of Internet & Society, which has just launched an intriguing new experiment called MediaCloud. In an interview with Joshua Benton of the Nieman Journalism Lab, Berkman Fellow Ethan Zuckman describes how MediaCloud works:

“It is a very large set of data, as well as some simple tools for playing with it, obtained by subscribing to and processing hundreds of American blogs and a couple hundred newspapers in English from around the world.

“The idea behind this is that we subscribe to the RSS feeds of these newspapers and these blogs. We grab every single story that they publish. We then pull the story text out of the HTML, which is an interesting hack. We throw the story text into a bunch of different tools that help us determine what the stories are about. So we’re able to get topic information. We’re able to get information on people mentioned in the stories — what’s called named entities.

“And then we file this all off in a database. So if you then want to find out what the stories were on Fox News for a given week, we can tell you what their top-10 topics were. We can also go levels further and say: When a news source reported on a topic, what other topics were most closely associated with it?”

Playing around with MediaCloud this morning, I can see that it is very much a work in progress, but that its upside is tremendous. Comparing the top ten terms associated with the key word Google in The New York Times, the San Francisco Chronicle, and Pajamas Media (a blogging site where I occasionally post media items), I obtained the following results:

• The Times most frequently mentioned Microsoft, eBay, Yahoo, and a host of computer and electronics terms (as well as porn) in its coverage of Google, suggesting a definite business focus.
• The Chronicle most frequently mentioned places (as well as one person)– U.S., San Francisco, Washington, California, the White House, Barack Obama – suggesting more of a political and geographical focus in its Google coverage, although Yahoo, Microsoft and YouTube did make the list.
• Pajamas had an almost entirely different list dominated by the topics its bloggers obses about — Iraq, Hamas, Israel, Gaza, food, Republican Party, Obama — indicating Google’s main function here was as a search tool.

Mine is quite a primitive analysis, but you can see some of the potential comparative value for professors, students, and analysts following the media industry, old and new alike. What’s perhaps less obvious is the opportunity for new media businesses.

As a new media company emerges, part of its winning strategy will have to be to define those niches in media coverage where it might expand its audience most rapidly. MediaCloud may emerge as the kind of data analysis tool that helps entrepreneurs do just that. There are others — Google’s News Trends for one — but this is still underdeveloped territory, worth monitoring closely as it develops.

As my Bnet colleague Erik Sherman notes in his post on MediaCloud today: “…the real value the cloud is going to offer is the ability to combine information from different sources and gain new levels of understanding in such areas as strategic planning, operations, or market analysis.”

Thanks to Tamara Baltar and Erik Sherman for pointing me to today’s topics.