Transcripts: AEJMC 2003

Jan Schaffer, J-Lab Executive Director: Welcome. As many of you know, J-Lab is a new center at the University of Maryland’s College of Journalism. It is a spin-off of the Pew Center for Civic Journalism. Its mission is to explore what the potential is for new technologies to help engage people in public issues.

“We tend to think in terms of news experiences rather than news stories.”
-Jan Schaffer

We tend to think in terms of news experiences rather than news stories. To help ferret out what those innovations might be, the Knight Foundation has funded us to launch a new Batten Awards for Innovations in Journalism in honor of Jim Batten, a former Knight Ridder CEO who was a pioneer in connecting with communities. I want to thank the Knight Foundation.

I also want to especially thank the Ethics and Excellence in Journalism Foundation, which supported this luncheon through a grant to J-Lab.

Today, we have a wonderfully innovative panel of journalists and, I tell you, I did not know this when I invited them, but they were all finalists or semifinalists for this year’s Batten Awards. The winners will be announced Sept. 15 at an interactive journalism symposium at the National Press Club in Washington, DC.

Eric Pryne is a long time reporter for the Seattle Times. I first knew him when he spearheaded The Front Porch Forum, which is a very long running civic journalism initiative there. He really has devoted his career to reporting on public policy issues, things like growth and transportation. For three years he was the watch bureau chief for The Times, he has won numerous regional and national journalism awards. He was recently a Stanford Journalism fellow. He will be talking to you about a project to deal with the overwhelming gridlock in the Seattle area right now. It is called You Build It.

After that Mike Skoler will take the stand to talk to you about a whole new initiative that Minnesota Public Radio is starting to do. Mike has had a very interesting career. He left the French wine business to join journalism after reading a book called How to be a freelance writer. And he is now managing director of Minnesota Public Radio. His charge is to develop a whole new model of news, hopefully they can take it NPR wide, one that consistently taps the smarts of the public to inform the journalism. He is formerly a science writer, a foreign correspondent for NPR, and a Nieman fellow. He will be talking to you about a Budget Balancer exercise that was a Batten Awards finalist and he will also discuss what MPR is trying to do overall in its interactive initiative.

Angela Clark tell you about what she thinks is the most sophisticated project MSNBC.com has done to date, called The Big Picture. She has been, and I love her title, journalist/technologist for 16 years, she is the director of interactive content for MSNBC.com. She previously worked with WashPost/Newsweek Interactive in Rosslyn, VA. Before that she was a copy editor, news editor and wire editor. She is charged with leading the interactive producer team at MSNBC.
I want to be sure to leave time for questions at the end so II’d really like to get moving quickly and start with Eric. Thank you.

Eric Pryne, Reporter, The Seattle Times: Thank you, Jan. After we unveiled our “You Build It” project last spring (2003), one reader who wrote to thank us for it quoted a Chinese proverb that I think captures the power of interactive journalism pretty nicely: “I hear and I forget. I see and I remember. I do and I understand.”

New technology now allows newspapers to interact with readers both more easily and more intensely than we could before. It allows us not only to tell or show readers news, but to give them the opportunity to do news and perhaps understand it better. We’re just beginning to explore that potential.

“I hear and I forget.”
“I see and I remember.”
“I do and I understand.”
-Eric Pryne

“You Build It” appeared in The Seattle Times on Sunday, March 30, both in print and online. We invited readers to play the part of regional transportation decision makers — to do transportation policy, not just read about it. We presented them with many of the same difficult choices real world decision makers face.

We wanted to explore a new way, and maybe a better way, to inform readers, but we had another goal: We also wanted to empower readers, to give them a voice in an important regional dialogue. This was really a civic journalism exercise, and the results exceeded our expectations. Nearly 2,000 readers participated in the 2 1/2 weeks we allowed, and many of them told us they understood the trade-offs and complexities much better than before.

The project also had an impact. After the readers made their choices, we did stories on what they wanted, and, in some instances, their priorities differed from those of the regional decision makers. After we wrote about that, the decision makers began to move in the readers’ direction.

Some real world context: When “You Build It” was published, a new three-county board was piecing together a regional transportation package to submit to voters. At the time, they were looking at an election this fall. They had made some tentative, preliminary decisions on what taxes to raise, but they were deeply divided on what projects to propose. Some wanted more transit, some wanted more highways, and they hadn’t come close to finding enough money to build everything they wanted. They also hadn’t done much to involve the public.

The Times had been writing traditional news stories about this agency’s deliberations, but we thought an interactive exercise would be a fun way to involve more readers in the issue. “You Build It” presented voters with a giant ballot of transportation projects and taxes, and asked them to design their own package. We asked them to pick the projects they thought should be built and then find the money to pay for them, just like the real transportation planners. We promised our readers that we would tabulate the results and report on them; that this would be a two-way street.

Each project on the ballot –let me scroll down here to give you some idea of what we’re talking about — included a thumbnail description of the project. For some of the more complex mega projects , we gave people a number of options, including a thumbnail description and a price tag, and people could click on the option they wanted and then scroll down and go farther.

We divided the choices into three categories, the so-called mega multibillion-dollar projects, other road projects, and transit projects. At the bottom of each category we gave a running total of what their choices were adding up to.

Each of the revenue sources included an estimate of how much money the tax would generate over 15 years. We asked readers to try to come up with enough revenue to pay for the projects they wanted — to balance the budgets.

A game like this probably allowed us to present information in more detail, especially on projects, than we would have in conventional news stories. We could provide more detail because people were going to actually use the details. We could do some things online that we couldn’t do in print. For one, online the calculator did the math for people and told them how much they were spending; they didn’t have to pull out a calculator or anything. See, right now I have something like $19 billion worth of projects and no revenue so… (laughter)

“New technology now… allows newspapers to interact with readers both more easily and more intensely than we could before. It allows us not only to tell or show readers news, but to give them the opportunity to do news and perhaps understand it better. We’re just beginning to explore that potential.”
-Eric Pryne

We also had a pop-up box in the online version that came up when people punched “submit” that told them how close they had come to balancing their budget and then asked them if they wanted to revise their plans. We didn’t require balanced budgets, but most people came pretty close. On average the people who participated online actually ran a small surplus, which is definitely not real world. (laughter)

We decided early on that with many older readers who probably didn’t have Internet access, we couldn’t just offer this ballot online. We thought we had to let people respond in print. However, of the nearly 2,000 readers who filled out ballots, about 85% of them participated online, and it’s easy to see why. Filling out the ballot online was much less labor-intensive and time-consuming: People only had to click their mouses. But people who filled out print ballots had to pull out a pencil and a calculator, do the math, clip out the ballot, fold it up, put it in an envelope, address the envelope, put a stamp on it and put it in the mailbox.

So we got many more participants by going online and we also probably reached a different audience. We didn’t ask many demographic questions on the ballot. We figured people would have limited tolerance for too many personal questions and that might affect participation. But from the few questions we did ask, we found that online participants, on average, had lived in the Seattle area fewer years than the print participants, and if you assume people who have lived in an area for a shorter time are more likely to be younger, then maybe we were reaching a segment of the population that newspapers are desperate to reach.

Putting this exercise together wasn’t simple. We wanted to make it resemble the real-world, policy-making process as much as possible, but we also wanted to keep it from getting so complex it would turn readers off. Also, there were lots of variables and we knew we couldn’t account for all of them. For instance, one political reality that the real-world transportation planners have to consider is geography, making sure every part of the region gets something. We decided requiring that kind of equity would make our exercise too complicated, although we did ask readers to consider it.

We had to make some fairly arbitrary decisions about what projects and revenue sources to include, and what estimates to use. We put the ballot together after consulting extensively with government agencies and interest groups to let them know what we were doing and seek their ideas. All of them were very willing, even anxious, to help us. We didn’t do everything all of them wanted, but none of them accused us of stacking the deck against them.

We were pretty transparent in explaining our process to readers, as well. We included a sidebar in the paper and online that explained how the ballot was put together, and we didn’t get many complaints. We ended up trimming that sidebar for space in the print version; we didn’t have to do that online. We also involved The Times’ public opinion research firm in our planning. We knew this couldn’t be a scientific poll because the sample would be self-selected, but we still wanted our pollster’s help in designing a ballot that would be fair, and we wanted his help later in tabulating and analyzing the results.

Now, reporting the results of this required some special care. We couldn’ t report on it like we would a poll. We knew from the demographic questions that we did ask that our participants weren’t statistically representative of the region as a whole, but we found no evidence that any interest group had stuffed the ballot box. And we felt the results were still worth reporting because the people who participated actually had spent some time pondering the issues and the trade-offs, making choices. They were informed, which isn’t always the case with people spouting opinions in quickie scientific telephone surveys.

” ‘You Build It’ presented voters with a giant ballot of transportation projects and taxes, and asked them to design their own package. We asked them to pick the projects they thought should be built and then find the money to pay for them, just like the real transportation planners. We promised our readers that we would tabulate the results and report on them; that this would be a two-way street.”
-Eric Pryne

We didn’t require it, but about half the 2,000 people who participated in “You Build It” gave us their names and phone numbers or e-mail addresses. That’s a much higher percentage than we get in our regular polls when we ask people if they would like a reporter to call them later. Those 1,000 or so people who gave us their names and e-mail addresses are a valuable resource that we may use in follow-up exercises, and there’s plenty of time for that now; the regional transportation board has put off a vote until November of 2004.

We’re just beginning to scratch the surface of this, and this is just one of several interactive journalism exercises The Seattle Times had done. I do have a few copies, about a half-dozen or so, of the project as it appeared in print as well, if anyone is interested. Thank you (applause)

Question:
Who designed your online form? Was it in house or…?

Answer:
We did it in house, and I think if we were to do it again, we would do it a little differently. Scrolling through a long ballot like this can intimidate some viewers and readers, and I think we would have maybe broken it up into more pages. I think we also, let me see if I can, this is not my computer, as you can probably tell by now. We also had a map that people could click on to see the locations of these projects. (there we go) and I think if we were to do it again, rather than having a big map of the whole region like this, we’d put it together in smaller chunks so that it could provide more detail.

Question:
Can you still see the exercise online?

Answer:
You can look at it, yes. It’s been disabled, so you can no longer enter anything. For instance, the pop-up box that asked people to balance the budgets is no longer something I can pull up, but you can look at it and still click on things and the calculator will still work. The url is: http://www.SeattleTimes.com/trans_budget/

Question:
Did you do any sort of usability testing to see how the citizens were using it: What worked what didn’t work?

Answer:
No, we didn’t. We were just sort of delighted that, with the response we got, that we figured enough people had found it worthwhile. I remember talking with our pollster beforehand. We were saying, “Wouldn’t it be great if we got 1,000 people to participate in this,” and the participation was nearly double that, which again shows the power of the Internet for us to engage in dialogue with our readers much more easily than we ever have before.

Question from Jan Schaffer:
Talk about the impact on the transit board.

Answer:
Yes, one of the things we found is they were looking at a tax package that was pretty dependent on sales tax increase, and when we presented a number of revenue options — realistic revenue options to readers — we found that they were much more inclined towards gas tax, vehicle taxes, and only a quarter supported a sales tax increase of the magnitude that the board was considering. I interviewed most of the board members after we reported the results and they said, “Yeah we probably need to go back and trim the sales tax and look at increasing the vehicle taxes.” Now whether or not they do that remains to be seen; they still have more than a year to play with this.

Question:
I was just wondering how you had promoted this.

Answer:
We had a tease to it on the front page of the Sunday paper, we had another more extended tease on the local cover. The package itself in the paper filled up two pages inside the local section. We promoted it in the newspaper several times in the 2 1/2 weeks we allowed people to vote. And on the Sunday, we also highlighted it on our main seattletimes.com web page as well and kept it there in a prominent place for the duration of the exercise.

Question:
Did you say earlier that you did not get the ages for the participants?

Answer:
We did not get the ages, no. We figured that if we asked the demographic questions that you would normally ask in a telephone survey, people would shut down; it would be a turn off. So we asked where they lived, what part of the region; we asked how long they had lived here; and what their principle form of transportation was. That’s it. We figured those were the most important for our purposes.

Jan Schaffer:
We’ll take more questions at the end. Let’s move on to Mike Skoler. Thank you , Eric, that was great. (applause)

Mike Skoler, Managing Director of News, Minnesota Public Radio: In the interest of full disclosure, I should mention that The Seattle Times earlier work in interactive journalism was one of the inspirations for what Minnesota Public Radio is doing.

I came to Minnesota Public Radio seven months ago with a mandate to invent a new way of doing journalism that brings the audience into the news process. We call it interactive journalism, but, when we use that term, we mean it a little differently than simply presenting information in a way that directly involves the audience and requires audience participation.

At Minnesota Public Radio. we create simulations, we do games, we do surveys, we use online tools to engage and converse with our public, with our audience, but those are tools, even if important tools. At Minnesota Public Radio, interactive journalism is something different for us. It’s about gleaning knowledge and insight from the audience. It’s about improving our daily journalism and the way that we actually set the news agenda in an attempt to be more of a leader of news coverage in our region.

We start with the assumption that, on any given story, there is someone in the audience who knows more about it than we do. That might be a business leader who won’t go on the record; it might be a middle manager who would be hard to identify in the organization; for a story on adolescent mental health, it might be the parent of a teenager who’s mentally ill. Our goal at Minnesota Public Radio is to tap into that audience expertise and do it in a way that is large, engaging thousands of people, and directly impacts the journalism that we do, the quality of the journalism and the way we make story selections.

“We start with the assumption that, on any given story, there is someone in the audience who knows more about it than we do.”
-Mike Skoler

It sounds noble, A, but if you’re the skeptical sort, and I’m assuming you are, since you’re journalists and educators, you’re thinking, “How do you do this without swamping a newsroom with information from tens of thousands of folks? And how do you do it without loosening the standards of journalism that we all strive to maintain?”

I’ll admit off the bat, we’re not fully sure how to do that, and that ’s what we’re trying to develop and certainly doing in our series of experiments. What I’d like to do is talk about some of those experiments, but mainly describe our most ambitious one to date, which is the MPR Budget Balancer.

Minnesota, like many states, had a huge looming deficit, $4.2 billion, and it was a major regional news story throughout the spring. Budget debates are tough to cover, I’d assume some of you have had experience with that, because budgets are so big and complex, that what you tend to do with a budget is break up it into pieces, and you talk about how spending cuts or extra spending might affect health care, might affect highways or whatever. It’s really hard to present the budget as one whole piece and really help the public feel the tough trade-offs that come with trying to set a budget, especially in the current environment where it’s all about reducing deficits.

“At Minnesota Public Radio. we create simulations, we do games, we do surveys, we use online tools to engage and converse with our public, with our audience, but those are tools, even if important tools. At Minnesota Public Radio, interactive journalism is something different for us. It’s about gleaning knowledge and insight from the audience. It’s about improving our daily journalism and the way that we actually set the news agenda…”
-Mike Skoler

So we wanted to know both how people balance budgets but also how they think through the budget trade-offs. Our aim was to engage lots of people in this and, by figuring out how they think through the budget, actually pick stories that would be most meaningful, most important to our community. So we created a simulation, and called it the MPR Budget Balancer. I won’t show you all 19 screens of the balancer, which provided people with 62 different choices on how to balance the budget, both cutting spending and adding revenue. It takes about 10-15 minutes to get through it, and we were really concerned whether or not people would stick with us that long.

This first screen was kind of a setting of expectations and a bit of a warning that it would take a little bit of time and that they didn’t have complete freedom in this: There would be a specific set of options that we provided, and they would have the opportunity to present others later.

We required an e-mail and a zip code. We thought about not requiring that to try to get the most people in, but we really wanted this to be a bit of a transaction where, in order to play with us, we wanted to find a group of people that would engage with us regularly in our news coverage. We asked for some demographic information, and we were pleased to find that 98 percent of the respondents — this part was voluntary, the demographic information — 98 percent gave us some information and more than half gave all of the information we asked for. We asked for simple things: We asked for age; we asked for where you lived, whether it was city or suburbs or out in a rural area and the like; what gender you were.

After giving information about the current budget, we offered users some big swipes at the deficit. For instance, Option 2, which you probably can’t read, you could start with a common political move and use some creative accounting to delay payments on state expenses. This proved very popular (laughter) as you can imagine, and popular with the politicians, saving $550 million.

Then we offered specific budget cutting options by category for more than seven screens. Here’s health and human services, and each page, as you’ll see on the right, starts with headlines, which provide basic facts about the category and it might explain where Minnesota compares with other states in terms of spending in that category. If you choose an option, like the first one, to cut basic health grants for low income adults and families, two things happen. First, you’ll see the deficit clock, which is a little hard to read here, changes, it now says $3.6 billion instead of $4.2 billion, so you’re cutting the deficit, your goal is to get it to zero.

Secondly you get this big red box that says “Look Out” and talks about the possible consequences of the choice that you’ve made. In this case, up to 33,000 Minnesotans could lose access to state-subsidized health insurance. That’s the standard form for all of these options.

From cutting, through these seven screens, we move to options for adding revenue, and the pie chart explains the breakdown of where money comes from now and at any point you could click on a category, like sales tax, and see an explanation, in this case saying that the current sales tax is 6.5 percent for some goods but there is no tax, for example, on clothing. Then we offer two pages of revenue-raising options, so we might click on the first option to raise income taxes and & we find out that Minnesota might move from the eighth highest taxing state into the top five.

“So we wanted to know both how people balance budgets, but also how they think through the budget trade-offs. Our aim was to engage lots of people in this and, by figuring out how they think through the budget, actually pick stories that would be most meaningful.”
-Mike Skoler

When you balance the budget, you get a pop-up box that provides you with congratulations and gives you the option to continue cutting, if you want to actually create a reserve, or review your material, or to submit. At any point, on every page, you’ve got a submit button at the bottom, where you can get out of the simulation and go to the payoff, which was an immediate comparison with the governor’s proposed budget, line by line. The checks show where you agree with the governor, and where there’s no check you have a difference with the governor’s proposal. The governor’s proposal was really the working document for our budget debate.

We provided an easy-to-print version of this, which was very popular with secondary school teachers, and we’ll talk more about that in a second. Then we provided a visual look of how your plan affects the overall budget pie compared, again, with the governor’s plan. In some other screens, the balancer shows you how the state ranks on various quality-of-life measures, from educational achievement to level of taxation, and then asks people to consider how their choices on this budget might, over time, affect the state’s future and its ranking on those. I won’t show you that, for time.

Finally, people could tell us if they were surprised by their choices. Seventeen percent said they were. Many of them said they were surprised that they had voted not to raise taxes but found themselves raising taxes because of the trade-offs in this.

We provided the opportunity to comment on the balancer and its options, and one in every five people provided comments about the simulation. We recorded every choice every person made for analysis and that’s what made this different than just an interesting way for people to engage and learn about it. We wanted to learn from what people were doing.

The project took a lot of folks. We did it in 3 1/2 weeks, and it was tight, and things were changing on us. The governor’s proposal when we started wasn’t even on the table yet, the numbers changed through it. We ended up hiring a software firm in California with a lot of experience with educational software to actually do the total of this, because our new media folks just didn’t have time to do it and it was quite intensive, but we consider it highly successful.

Numbers are higher than what Eric Pryne was talking about. We had the advantage that we had radio to actually promote this and we promoted it heavily. After every budget story we did, a host would announce, “Go to the MPR Budget Balancer, you take a crack at the budget,” so we were able to engage a lot of people; 19,000 checked out the balancer, which for us was a huge response.

People wrote comments thanking us, many organizations promoted the balancer or linked to it on their sites, such as the Minnesota League of Women Voters. A county commissioner in a public hearing actually stopped and urged all of his colleagues there to go on the MPR Budget Balancer; you couldn’t ask for better publicity. “Come create your own budget. You come to us with your plan.”

We ended up with 7,000 people who actually stuck with us for the 10-15 minutes or longer, and the 7,000 people actually generated 11,000 plans. Many of them actually submitted more than one plan. We got 7,000 by removing the doubles and by removing people outside of the state and the like. One expected, but still very nice surprise, and Eric referred to this, is that we attracted a somewhat different audience. On the right you have the typical public radio listener and on the left the Balancer users. Generally just 10 percent of public radio listeners are 30 or younger, 43 percent of the balancer users were 30 or younger. Obviously the medium has something to do with that — computer use.

Also we reached a lot of school-age kids. 0-18. It says 17 percent, I think the final tally was actually 20 percent. We promoted this heavily to secondary school teachers; many of them assigned it for extra credit and assigned it in class. And [we reached] a large number of 19- to 30-year-olds. We feel this fits with the notion that there’s a new generation of folks who are used to being engaged with nearly every aspect of their lives, whether it’s planning their finances and trading constantly or whether it’s actually interacting with news and public information and being a part of the news that they receive.

And the Balancer strengthened our journalism. From user comments and the popular choices that we saw, we got an outside perspective on our coverage and a sense of some potentially important issues that weren’t being covered. A popular choice for cuts on the Balancer was a health care rationing system, similar to what Oregon had done, which, instead of cutting the rolls of who could receive health insurance, it cut some of the less cost-effective services; it reduced services. So we did a little story explaining that before the issue was actually proposed in the legislature.

“Finally, people could tell us if they were surprised by their choices. Seventeen percent said they were. Many of them said they were surprised that they had voted not to raise taxes but found themselves raising taxes because of the trade-offs in this.”
-Mike Skoler

Taxes were another big issue: 85 percent of the people who participated actually ended up using the revenue-raising opportunity. As Eric mentioned, this is a survey not a poll, so we went out and commissioned a poll to see if what we were hearing, what we were seeing in the Balancer, was actually an undercurrent that wasn’t being recorded. What we did in the poll was we provided some of the trade-offs, and we found that when we asked people in Minnesota, “Would you cut taxes or would you raise revenue?” in general the majority said they don’t want taxes raised, let’s cut spending. But when we actually presented specific options, we found that the majority of people actually would raise taxes in order to preserve some services, up to 70 or 80 percent, and we actually found that 70 percent favored two specific ways to raise taxes, so we were able to actually write that story thanks to the Balancer. A number of stories came from it.

This wasn’t the end of the story. Through this project, we ended up with nearly 5,000 people who said they would engage regularly with us in helping us with news coverage. During the Balancer process we did a series on eating and overeating, on obesity, and we got about a 2 percent response rate from folks, which seems small to us, but was helpful, People shared some very personal stories, and I’m not going to play the CD I was going to play for that because I’m running out of time.

Since then, we’ve been doing surveys of this group of 5,000, and we asked them a follow-up question at the end of the budget debate. This time 18 percent of them, 860 of the 5,000, actually came back and gave us detailed reactions. This is an example of the report we get, not the survey, and we blacked out the names at the top, those aren’t bars of percentages. Each question we ask would get a response and we would put those into reports that we would then provide to editors and reporters doing stories. In some cases, we went back to specific people who had information that identified that they had expertise; we used them in stories.

“Generally just 10 percent of public radio listeners are 30 or younger, 43 percent of the balancer users were 30 or younger.”
-Mike Skoler

We also currently have a survey that’s being used nationally, as part of a collaboration on public radio stations and web links for special coverage appearing this November. The series is called (inaudible), the coverage is called “Whose Democracy is it?” We have this survey up, and this actually resides on station web sites, and now National Public Radio has it, and when people fill out this survey, and it asks them questions like, “Do you participate in sustaining democracy,” “Give us a story of when it’s worked for you or hasn’t worked for you,” in fact it goes over a national database that we’re using to help guide the editorial material for the special coverage, it goes back to the individual station the user kind of came through. So local stations are getting a sense of what’s important in their local area, we’re getting a sense nationally of what stories and source ideas are nationally. We’re using this very regularly, this interactive survey tool that we’ve created. We’ve kind of taken where we’ve gone from the budget balancer, we’ve taken that group of people and we’re now in regular contact with them, on this and other things. We plan some other group projects, but with these its constant interaction we’ve got.

Question:
(inaudible)

Answer:
We didn’t seem to, but we were very clear from the start that we didn’t want to publish any of these results, and we never spoke of percentages of people who chose anything because we just wanted to make it clear that this was to help inform our judgment and our insight and our editorial sixth sense, it wasn’t to actually report on this and that’s why a key part of what we do is, if we have a hypothesis based on this we create a poll to actually get some scientific results. So we didn’t sense that and there was no agenda, there was no reason to kind of come in and skew this because it didn’t appear on the air. Yes?

Question:
(inaudible)

Answer:
What we’re doing is, we’re promoting it on air, we’re promoting it on our web site and with some of these e-mails to these folks to alert them that there’s a new survey up. What we’re trying to do is actually keep building that base. It’s not just about technology, we actually have a salon program where we’re going to start going to peoples homes to have discussions around interactive journalism & Minnesota state fair is coming at the end of August, we’re going to have a booth there where people can come in and talk about a particular question for a minute or a minute and a half to get out the idea that we’re trying to listen, we’re trying to gather people, as well as people not connected to a computer with an 800 number and the like, so we try to keep building that, but this base that we’ve got is a base that we’re getting information now about occupation and the like so that we can actually send out selective e-mails.

Question:
There’s a lot of content on this site, how did you create the content, how did you verify…(inaudible)

Answer:
A lot of hard work, and that’s why there’s that big team of people up there. What we ended up doing is, besides sending two reporters out full time to gather this and analyzing all the documents from the governor and documents from other organizations in the state, once we had an initial draft of the Balancer we started testing that among experts and people of different political parties to try to catch where language seemed to be biased or facts were in dispute and then, as good reporters, we would go out and actually try to determine our best sense of what the true facts were about impact or  there were even disagreements about the governor’s proposal and what it actually did that we had to sort out, it’s just a lot of basic journalism. It’s scary, because you’re putting out a lot and the ability of people to attack you is great, so there’s a concern. Thank you. (applause)

Angela Clark, Director of Interactive Content, MSNBC.com: I hope you had enough of your dessert so you can be a little bit awake. I’ll try to keep it exciting.

I’m from MSNBC. We’re a partnership between Microsoft and NBC and so we are lucky to be able to have a lot of traffic, some experience in journalism from NBC, and some experience in technology from Microsoft.

MSNBC has been out there for seven years now, and we’ve been committed to interactivity that entire time. We’ve been really trying to learn lessons, trying to figure out better ways to do interactive journalism both in small and large ways. We make sure that we have interactives on all of our major news events. We make sure we have unique coverage of an ongoing story by having interactives give depth, breadth or context to the story, hopefully all three in one take.

“The No. 1 most important thing, even though it’s last, is enticing the reader to experience, to think, and to engage themselves into the news.”
-Angela Clark

It also gives our staff a chance to flex their creative muscles, and sometimes we go overboard with that. We also experiment with new storytelling techniques, and we’re constantly checking to see what works and what doesn’t, using a lot of both qualitative and quantitative information and I can explain that later.The No. 1 most important thing, even though it’s last, is enticing the reader to experience, to think, and to engage themselves into the news.

With us, we basically have two choices when you first look at a story: What kind of interactive do I want? Is the story itself ongoing? Do I want something that is “referential,” that is basically going to give a piece of information about the story that, if we put it in the story, in the text itself, it detracts from the story but you still need to have it. You know, in the text days… where you could type in a paragraph describing that long-running story of “Monica Lewinsky was an intern…” and blah blah blah. Now, we can use our interactives in that way, so those referential pieces give historical context, lots of information about a topic, or it gives a sense of why this story is important and why we’re playing it so well.

So we’ll get something that is, like, a small self-contained jewel, let’s say, a piece of information. For instance, like this, that we had way back when where we had something all about Ephedra. Now this is something that could go into a story, and it was templatized, yet it provided a lot of information that we didn’t have to explain every single time, and gave more depth and breadth to the story itself. Now because it’s a referential interactive for us, we make sure that we templatize these things, so that Ephedra interactive you just saw was made in about 5 minutes. The longest time was to create the text and just type it in. So we try to make sure that those types of interactive are easy, quick, and that the editors aren’t in any way encumbered by the technology behind it and can then concentrate on the journalism part of it.

As we’ve gone on in our quest for better interactive journalism, we’ve found there’s another approach to interactivity and that’s the “narrative.” That’s the one I’m going to be talking more about here. That’s where we tell the story to the reader; we have an alpha and omega to the story itself. Whereas with a referential, somebody may want to find out more about why Ephedra is important, but they don’t care about its effect on the body. With a narrative interactive, we have a specific story to tell, we want to make sure you hear it all the way through, and we want to make sure that we use all of the fabulous technology and resources that we have available.

We’ve also found that, every time you ask somebody to click on something, you lose about 50 percent of your audience, so we try to make sure we don’t ask people to click for more information. You’ll notice a lot of our stories kind of give people that “lean back” experience, similar to what you have in a television show. At certain times, we try to make sure where we re trying to get somebody to be engaged or think… that’s when we ask them to click, because we’re asking them to make that decision and we want the people who are engaged to continue on, so I’ll explain all of that.

“At MSNBC our audience is at least 51 percent broadband and it’s increasing by about 10 or 15 percent every year”
-Angela Clark

“The Big Picture” is our latest in a series of approaches to interactive storytelling. It is basically the latest of everything we’ve learned from all of our interactives, so I’ll walk you through some. I’m hoping that you guys might have seen one of these before… You’ll notice that we specifically ask people to launch this, we’re telling them to adjust your volume and all this, because when you do an automatic pop-up, people in this day of pop-up ads will automatically kill those out. “The Big Picture” also requires broadband, and it also requires Flash 6. This is a very specific decision we made because at MSNBC our audience is at least 51 percent broadband and it’s increasing by about 10 or 15 percent every year, so we’re choosing to just do this.

We do some detection automatically, so that when you come to this and you’re coming from a dial-up or you come without Flash it will say, “You need a broadband connection for this,” or “You need Flash.” So we ‘re already cutting off a big chunk of our audience, but it was a very conscious decision on our part.

So here, with the conflict with Iraq… (pause) it was, it was (laughter) Let’s try this one more time, huh? There we go.

Audio runs:

Lester Holt, Big Picture Anchor:
“The ultimate weapons. A nightmare for a nation rattled by terrorism. After a decade of defiance, Saddam Hussein says he has nothing to hide. President Bush wants proof. And while many of America’s allies hope for peace, it may come to war. The risks are enormous and the issues complex.”

Dara Brown, Big Picture Narrator:
“Tackle the tough questions. Can inspections work? If not, should the U.S. go it alone? Would war spin out of control, and is it worth the risk? Get the facts, go in depth, and decide where you ultimately stand. I’m Dara Brown.”

“And I’m Lester Holt. This is ‘The Big Picture.’ “

“We want to make sure that we give the readers control.”
-Angela Clark

I’m going to go ahead and put this on pause as we continue. We’ve done three of these [Big Picture installments], and there are reasons why we do things the way we do. As we create each new one, we tweak and refine each time, but there are certain rules that guide us: We want to make sure that we give the readers control. So we have clear exposed navigation over on the left -hand side so that people understand the rundown, and what they’re going to be listening to, so if they really are not interested in some elements of what we’re telling them, they can control that experience. We have both the lean-forward and that lean-back experience. We give people control; if they want to click through and do what they want to do they can do that, or they can just lean back, and we’ll walk them through all the way.

For example, in our Big Picture of the Oscars, we found it very interesting that about 20 percent of our readers completely skipped the best actress element and went directly to the best picture. I don’t know why; maybe we had a large female audience or something and they didn’t care about the best actress but really had an opinion about the best picture, but it was very interesting. We make sure to track all those elements.

We try to make sure that we have that menu over on the left-hand side as a standard feature of every story we tell. We want to make sure it has text that looks clickable, that is underlined in some way. We also use cues to help people remember elements, to encourage them to read or to use it.

For example, if I go back to Big Picture, we use what we call our “giant finger of interactivity,” just let me see if I can get this here (pause)

Dara Brown audio: “Starting with commercial and spy satellites snapping pictures like these over Iraq. Imagine looking at an area twice the side of Idaho through a straw…”

We’ll just go ahead and skip through this a little bit. We’ll have the cues here, in the right hand side, you’ll see it says “time to vote.” This gives people the idea of, you know, you have only two more seconds, enticing them to go ahead and vote, but we’re going to move on with it. So if they choose not to participate, choose not to have control over their experience, we’re just going to move on, because this might be something they’re not specifically interested in.

I’m going to speed up on it and not go through all these examples. Each one of these show some experiments we did on this control aspect. We’ve seen that in some cases it worked and in others it didn’t, and that’s why we’re doing what we’re doing.

So, as for depth. We want to make sure we give context, directly, where it’s most relevant. We don’t just want to give people way too much information. So we have a lot of different approaches we’ve devised, or stolen from other people — things where you basically can opt out, where the presentation itself completely stops until the user makes a choice. We’ve found, as I said, we can lose 50 to 75 percent of users doing that, but sometimes it’s very important that you make them make those decisions at those times.

Then there are the opt-ins in which you’re pausing and encouraging users to be part of this, but it sometimes is difficult for them to see. For example, we’ll go back to our big picture again… I’ve got too many Big Pictures going on… In this case & we have a piece in which we’re talking to Ken Allard, one of our experts & right here, where it says “Click to ask Mr. Allard.” Now it’s very subtle but it allows the reader, if they want to find more information about that particular piece of content, or what Mr. Ken was saying, they can specifically click on that and find out more about it. We’re not disrupting the storyline of what we think is most important, but we’re allowing them to explore and to delve into this more. This is one of the techniques that we use.

Finally, we have the supplemental. One of the criticisms we’ve heard on this, and honestly I have a problem with it as well sometimes, is that, we have a lot going on at the same time here. We have Lester, with his talking head down there, we have other things going on, and you’ll notice, down at the bottom it says, “Related: hear how people in the Middle East…” and so on. We found that as long as we have that set aside, it can’t really be too much information, it has to be less than X number of characters or words, and we want to “save for later,” because asking people to stop and explore that at that particular time is just not going to work and hasn’t worked for us.

So in this case you hit “save for later” and it goes up over on the left-hand side, so that they can see those links later. We also offer the option that, when they leave this particular Big Picture, it’ll say, “Do you want us to e-mail all of those that you saved for you?” so that those people can go back and learn more about that particular topic. That’s something that about 20 percent of the people who saw this clicked on and actually used. So it was a very effective element.

As for introspection. We want to make sure we challenge the reader to think about the issues, so we have a survey system that we use all the time, voting and so on. It’s not in any way scientific. I’m sure all of us who have used surveys on the Web have found that there are people who know how to hack them if they have enough time. But it does allow people to express their opinions, and we try to make sure that we place it in such a way that, if we want people to listen to the facts first and then make their decision, then we always place the vote at the end. If we want to ask people their opinion, and then give them more information, maybe to disprove or encourage or tell them they did a great job, then we put the vote at the beginning. If we’re taking their pulse, then we just basically make the entire segment the vote itself.

Quite often we try to dissect it into smaller points, because these big pictures can last up to 20 minutes, and we want to make sure that people spend time listening to the information, and then we ask them the questions. But we don’t get too deep into “I’m not really sure,” or “I don’t really feel that that’s correct,” or getting too deep into the questions. We want to make sure that we keep them there so sometimes we’ll show the answers to the vote way at the very end, like we did on the Oscars, because that seems to keep them there a lot longer. We’ve kind of finagled a lot of that

And finally, with the community, we want to make sure we connect users with the story so we make sure that they become part of the presentation. We always have “e-mail this,” “write this,” and that’s one of the most popular aspects of our site itself, but it’s something that works really well with interactives. We’ll read our viewer e-mail and incorporate it into the story. So if we take a look at that “Big Picture” again and continue on… I won’t have you guys listen to all the experts here. So, for example here, we’ve incorporated our reader mail in this, so that you get a sense of how people reacted to this and whether it was effective or not.

We also want to make sure people comprehend, because we’ve tackled three things: the elections, the Iraq war and Oscars, two of which are very difficult and challenging topics. We want to make sure that people are active, that they’re listening to it and they’re engaged. We found, when we did something on Enron, that the best way to really engage readers and to make your point is to make sure your audio is in sync with your text, so you say it, you write it, you have them vote on it later. That’s something that engages them a great deal. We want to make sure that we break up the audio tracks, we want to make sure that we ask them to form tasks that teach…

This is another project that we did…

Dara Brown audio: “…it’s the air traveler’s last line of defense against terrorists. The machines that scan our carry on bags are only as reliable as the people seated at the controls. Before Sept. 11, many of them trained just 40 hours, and often went months without seeing anything suspicious. Now the government is taking over, but the job remains the same: making out a few suspicious objects in a sea of everyday items. Is it enough? See for yourself.

You are about to take a two-minute shift screening carry-on baggage. The scans you will see are real, most will be innocent enough, but a few, like this one, have deadly potential. Your job is to flag suspicious items for your colleagues to investigate. Use these controls to zoom in for a closer look…”

That’s the giant finger of activity.

Audio continues: “…show color scans that display organic materials, like that found in explosives, in orange. You can stop the conveyor belt at any time, but take too long and you’ll hear about it from angry travelers (light laughter). When time is up we’ll grade your performance. Ready? Good luck.”

So you can stop this, zoom in and out, look at it in color if you want. This was something we set up as a game…

Audio of a traveler:
“Think we could get this operation moving a little bit faster here?” (laughter)

Traveler #2:
“I’ve been standing here for 45 minutes & can you believe that?

So this was actually based on a series of stories we did where we were actually examining, it was a year after Sept. 11, and trying to figure out if things had gotten better or worse with the airport security situation. In this case, we actually went to the airport screeners, where they trained them, got the pictures that they use to test each other, and we created the game so you could be in that place. We also tried to give you a sense of the pressure that those folks were under…

Traveler #3:
“I just can’t believe this. We’re going to be here all day.” (laughter)

So we tried to place people in the role of airport security officer. Interactively, we thought that this was a good way to present the information without actually forcing somebody to read a lot of… oh, there’s a knife… all right. After two minutes, it comes back and it says “you caught X number out of whatever.”We had a lot of discussions about how we were actually going to grade somebody or, how loose we were going to make this, and we decided that people still needed to get a sense of how well they did…

“Asking the readers to actually perform the tasks so that they could learn how difficult the job is, to be a baggage screener, is very effective.”
-Angela Clark

Basically for us, that comprehension, asking the readers to actually perform the tasks so that they could learn how difficult the job is, to be a baggage screener, is very effective. We’ve taken those kind of lessons from baggage screeners and tried to make sure we incorporate them into The Big Picture and I think that it’s been very helpful. For instance, we’ve had a lot of baggage screeners write us and say, “Thank goodness somebody actually understands what we go through.” We have other people saying, “I’m not going to be so mean to them the next time I go through.”

Finally, there is uniqueness. We want to make sure that we do things that only the Internet can do. For that baggage screener, you wouldn’t be able to have that in a newspaper, in a magazine or even on television, that’s only something you can deliver on the Web, and so we need to make sure that we think interactively to break out of the bounds of what we know already. I started out in the newspapers and I find myself being very newspaper-centric and text-centric, but it’s smart to have a lot of young folks who have experience in a lot of different media, so they’ll come to you and say, “Well, wouldn’t it be great if we could just do it this way?”

So we make sure that we ask questions of our readers, that we track who hits what when, that we do a lot of experiments as to whether a pull-down bar or a button saying “next” is going to be more effective. We make these little graphs for ourselves each time and spread them around the company to make sure that everybody learns the lessons so that the next one we do is hopefully going to be better than the last one. So basically that’s my spiel.

Question from Jan Schaffer:
How long did it take you to do the big picture, I mean it’s pretty pizzazzy…

Answer:
We’re very lucky that we have some folks who are kind of very multitasking kind of folks, so it was about two weeks.

Question:
And what was the audience, how many visitors did you get, do you know?

Answer:
Well, for instance, the Oscar package we had up about a week, and it was really heavily played on both the MSN front and on MSNBC’s front, and we had about 250,000 people come to it and experience it. About 20 percent went all the way through, I think it was an 18-minute experience, and that’s pretty amazing for us, you know, that many people is pretty fabulous, especially for that long, and broadband, and Flash, and all those things that are going to cut a lot of people out already.

Question:
(inaudible)

Answer:
Sometimes, yeah. Yes, some of it they got from NBC or MSNBC cable or CNBC, but, for example, the expert panel that I was going to show you are things where we specifically were able to track down those folks because MSNBC had them in house and we just asked them a series of questions and cut and spliced it as we could.

Question:
(inaudible)

Answer:
That’s a very good question. I led the team of only about four people, and three of them are journalists who learned technology, and one of them was a technologist who learned journalism. It’s fabulous to have that kind of mix, because the technologist will think of templatizing things that the journalist wouldn’t, but the journalist will figure out how to use that technology to better tell the story, so it’s a really good approach to have both types in your area.

Question:
(inaudible) During the 18-minute experience, is there any time to slip in a commercial message?

Answer:
Did my business-development guy, like, complain to you? Basically we do have & let me show you & we have something here that we call advertisement, in this particular case it was just basically a static ad, but we didn’t have any specific thing so, quite often, my team, we’re the ones out there experimenting, and then the ad guys kind of catch up later once we’ve templatized it. So we’re at the stage now where we have templatized it and they are selling it where we’ll probably, in the advertisement, it’ll be a streaming media ad, and they’ll be sponsoring it as well.

Question:
(inaudible)

Answer:
Thank you. Well, we’ve been a part of the interactive television realm quite some time with Microsoft and with NBC, and we’ve kind of gone back and forth asking them what’s the best approach. This is definitely a combination of what we’ve learned from those experiments, but at this stage we’re not quite talking with them about using this approach. I think that there are some elements of it that interactive television really hasn’t caught up to this yet, but it will probably in the next couple of years.

Question:
(inaudible)

Answer:
This was actually about the conflict with Iraq on its way up to the war, so we did not show much of that. That was the most difficult aspect of this, trying to figure out what we would show and wouldn t show. There was so much associated with the conflict and what was the best approach to this. I think we made some decisions that, if we were to do it over again, we would do it differently. So that didn’t really answer your question & For instance, I think some of the things that we did in regards to weapons inspections, we were only able to tell half the story because and that was mostly because we didn’t know a lot at the time and it was a very timely situation where it was important at the time but after a few weeks it wasn’t important anymore.

Question:
(inaudible)

Answer:
Not in this particular one. We have other elements in which we do show a great deal about the Iraqi people, but not this particular interactive, not in this particular Big Picture, yes.

Question:
Is there a URL that they can go to?

Answer:
Basically if you go to http://www.MSNBC.com/modules/bigpicture/ and then whatever the topic was — so Iraq, Elections or Oscars.

Question:
(inaudible)

Answer:
Well they want to make sure that & a large part of the big picture is it’s evergreen… so we have to be a little too generic about things sometimes, and quite often, because of that, we can’t maybe use that NBC spot that was submitted simply because it was too specific. But from a visual standpoint we don’t really have any problems; the conversion of the video –this is flash video — you degrade a lot of the quality, so obviously you don’t want to have, you know, thousands of people in the background and expect the HDTV look, but we didn’t really put that in that smaller space anyway.

Question:
A newspaper would chortle with glee if they got 20 minutes spent on back-to-back coverage. Yet you have done these exercises that take that much time just on one topic. What do you make of the fact that your users are spending that kind of time playing with these exercises? What’s it telling you? Is it telling you anything? Is it telling you anything about our traditional journalism? Is it telling you anything about demographics?

Answer:
Well, from the big picture, if you closed out of the interactive too early, it would basically say, “Why are you leaving?” and most of the time it was people saying, “I just don’t have enough time.” At least from the interactive standpoint. you’re getting people who are basically slacking off from work a great deal of the time, and you want to make sure it’s going to be information that they can go back to either later at night, when they go home, or something they can experience in bits and pieces throughout the day, and so we would fall to pieces to see somebody staying with it that long, especially since most of our traffic is during the work day,

Eric Pryne:
For a segment of our readers, admittedly a small segment, this gives them something that they haven’t been getting before. Our Sunday circulation is 475,000, we had 2,000 people participate in You Build It, if those 2,000 people spent 5, 10, 15, 20 minutes with the newspaper — and we had one respondent e-mail us and say, “You know, my family and I spent a couple hours discussing this around the kitchen table” — well that’s just the kind of thing that newspapers want, readers to spend some quality time and feel that we are essential to their lives and add something of value. This isn ‘t going to appeal to all readers, but those it does appeal to are going to feel a tremendous sense of appreciation and increased loyalty.

Mike Skoler:
I think there’s another element to it, as well. In a daily newspaper, a daily radio show, regular listeners or readers are actually following that story over time, so they get a fair bit of information over time, but they don’t necessarily catch everything. So, I think when you see people stay for 10, 15, 20 minutes on something, in a way it means you grabbed them. But in a way, I think that natural interest is there and that they can go back and get a big picture on a story that they’ve been following piecemeal and may have missed elements. So I think it’s encouraging in one sense because people feel very comfortable with this medium and spending time on it. It would be great if we could capture even more people doing that, and I have a feeling that societal trends are bringing us to a place where people are going to spend more time on specific subjects and less time kind of reading through front pages and skimming the tops of articles. We’ll see.

Question from Jan Schaffer:
These exercises are remarkably devoid of conflict, it seems. You don’t get a sense of scorecard journalism where it’s “Democrats vs. Republicans,” “Mayor vs. City Council,” “The Governor vs. the state legislature.” Any thoughts on the departure from that standard kind of framing?

Mike Skoler:
The Big Picture, actually, in some of that voting, kind of raises those issues in a sense. Tell us, do you support Dole/Gore… or not. I think one of the common threads here is that we’re tackling some of the bigger more complex issues, and we’re trying to provide more of a glaze than typical stories where there’s a clear conflict between a governor and a legislator or whatever. I think part of it is that these stories wade into stories that you only get a piece.