THE AGE OF DYNAMIC STORYTELLING

A guide for journalists

in a world of immersive
3-D content
The Age of Dynamic Storytelling
A guide for journalists in a world of immersive 3-D content

By Francesco Marconi and Taylor Nakagawa

Authors note:
This report on the next iteration of immersive media is comprised of two
sections. The first part captures some of the major trends and industry use
cases surfacing so far. These insights are derived from conversations with
dozens of leaders in the field of journalism, technology, academia and
entrepreneurship. For the second section, we collaborated with a team of
neuroscientists to conduct a scientific study measuring user engagement and
brain activity when consuming immersive media content.

The resulting ideas and observations allow us to identify ways that virtual
reality and 3-D content may impact the current model of storytelling as well as
its future development. We hope to present you with the key insights to make
your own informed decisions regarding how newsrooms can navigate the next
phase of immersive media.
The AGE OF DYNAMIC STORYTELLING | 1

Prologue
In the spring of 2015, two journalists in the interactive department at the
Associated Press received a padded briefcase with a heavy, rectangular-shaped
gadget. The object inside was a 3-D camera, a prototype provided by the
immersive media technology company Matterport, which combined infrared
and high-dynamic-range imaging to create “photo-realistic” virtual spaces.

A couple of months later, the two multimedia producers collaborated with a

colleague on the business news desk to embark on a project that would change
AP’s thought process on visual storytelling.

The trio set an ambitious goal to virtually transport news consumers to places
they would not — or could not — normally go, including first-class and
executive suites in planes, ocean liners and hotels. The resulting experiences Matterport’s PRO 2 3-D Camera (MC250) is a high-
formed “The Suite Life,” an interactive experience unlike anything our resolution 3-D scanning camera that The Associated
newsroom had previously produced. Press used to create its first virtual reality story, “The
Suite Life.”
“There are only so many times you can use phrases like ‘luxurious’ or ‘over the PHOTO COURTESY OF MATTERPORT
top’ to describe these posh accommodations,” said Scott Mayerowitz, the
business journalist who conceptualized the project.

“In fact, none of those words really do these unique spaces justice. But the
second people are transported into these virtual suites and can move around
on their own, their jaws drop in shock.”

Each space in “The Suite Life” was extensively photographed using the 3-D
camera and reconstructed for use in virtual reality. Using a VR headset,
viewers were able to “walk” through and experience each environment from a
variety of angles.

“We’ve found that the total immersion of VR makes the experience of

exploring a newsworthy location more emotionally impactful and ultimately
more memorable,” explained Matt Bell, a co-founder and the chief strategy
officer at Matterport.

The success of “The Suite Life,” helped AP journalists realize that storytelling
would soon evolve into a more interactive experience rather than being told
from a single, fixed perspective.

We call this new approach to journalism dynamic storytelling. In “The Suite Life,” participants had the ability to
virtually travel through some of the world’s most
AP is not alone. Innovators throughout the news industry are collaborating lavish accommodations. This 3-D model was captured
with technology companies and building their own studios to push the using Matterport’s scanning technology and shows the
envelope in a number of immersive media projects, redefining how stories are cabin of Singapore Airlines’ first-class suites.
planned, produced and distributed. AP PHOTOS

“VR is not going to replace traditional news formats, but it is an evolution of

our desire to want to get closer to the story,” said Molly DeWolf Swenson,
co-founder of RYOT, a production studio recently acquired by HuffPost.
The AGE OF DYNAMIC STORYTELLING | 2

As you will see in the following pages of this report, immersive media is much
more than 360 videos that provide a peripheral view of a scene. Emerging
technologies are on the way that will be capable of producing vibrant 3-D
models that give us an even greater sense of depth and texture. A technology
called “volumetric capture” will soon enable viewers to not only “walk”
through spaces, but also to touch 3-D objects and interact with others in a
virtual space.

“We experience the world with our whole bodies, so why shouldn’t we
experience stories with our whole bodies?” asked Nonny De La Peña, founder
of Emblematic Group, an immersive journalism studio in Santa Monica,
California. Virtual reality studio Emblematic Group leveraged the
volumetric scanning technology of 8i in order to
Broadly speaking, volumetric content promises to reap many rewards for produce “Out of Exile: Daniel’s Story,” a film that
journalism in the years to come, including higher engagement with audiences premiered at Sundance Film Festival in 2017.
and innovative reporting. PHOTO COURTESY OF EMBLEMATIC GROUP

“We experience
Of course, this wave of technological innovation follows a similar path from
others that have come before it. Each new technology reinvents how we

the world with our

experience news, activating each of our senses in a brand new way: The
printing press allowed people to read each other’s ideas; radio allowed us to

whole bodies, so
hear them; television allowed us to see them and immersive media is now
allowing us to feel as if we’re actually with others as they speak.

“The flat pane of glass with its two dimensions has been a constraint for why shouldn’t we
experience stories
content producers for decades,” said Justin Hendrix, executive director of the
NYC Media Lab. “With virtual and augmented reality technologies, that’s

with our whole

about to change. It’s hard to imagine the entire extent to which a new
dimension will create challenges and opportunities for journalists.”

To best leverage volumetric capture and 3-D content in news, the first step is bodies?”
to understand the technology itself. As you read this report, keep in mind that
all of the ethical and editorial considerations journalists have today will still NONNY DE LA PEÑA
FOUNDER
apply in the next stage of immersive storytelling. This analysis will both EMBLEMATIC GROUP
discuss those and hopefully inspire further dialogue about where these new
types of media could lead us.

3-D scanning adds a layer of depth to the pixels used

to produce flat photos and videos and, when captured
from all angles, volumetric capture can produce
holograms of people and 3-D models of places.
PHOTO COURTESY OF SCATTER STUDIOS
The AGE OF DYNAMIC STORYTELLING | 3

Trends and use cases

From video journalism to volumetric
capture
Before we explore the intricacies of dynamic storytelling, let’s first talk about
the transformational technologies enabling this shift.

First of all, there are some terms to get straight. In one, very basic sense,
this is all about the evolution of virtual reality. This new wave of technologies, Maya Alleruzzo, the AP’s chief photo editor for the
known collectively as “immersive media,” specifically include 360-degree Middle East, stands by her 360-degree camera during
video, augmented reality (AR) and volumetric capture (includes both CGI the filming of “House to House: The Battle for Mosul.”
and 3-D scanning). AP PHOTO

These techniques allow journalists to catapult audiences right into the middle
of the action, breaking physical and economic barriers by enabling
participants to travel to new environments, explore new realities and gain a
new perspective of the world around us.

Expanding perspectives
The first stage in immersive media was 360-degree video, or “360 video” for
short. This format gave journalists the ability to expand their frame of view
and capture spherical images.

“The era of 360-degree video is just beginning, but already we’ve seen its The New York Times’ six-part series “Life on Mars”
potential and begun to harness the strength that it offers our storytelling,” chronicled the lives of NASA astronauts living in
said Jaime Holguin, AP’s manager of news development. isolation on Hawaii’s Mauna Loa volcano to simulate
living on a foreign planet.
Below are three examples of how news organizations have utilized 360-degree PHOTO COURTESY OF THE NEW YORK TIMES
video:

–– AP’s “House to House: The Battle for Mosul,” brought viewers to the
frontlines of the battle against ISIS.

–– The New York Times’ “Life on Mars,” chronicled six scientists living in
isolation in Hawaii to simulate life on a foreign planet.

–– Sports Illustrated and LIFE VR’s “Capturing Everest,” gave viewers a new
perspective on the world’s tallest mountain.

In “House to House: The Battle for Mosul,” viewers are transported from one
location to another as they follow a group of Iraqi insurgents and experience LIFE VR and Sports Illustrated’s “Capturing Everest”
the difficulty of combat in a dense urban environment. followed the climb of Jeff Glasbrenner and his crew
up Mount Everest in a four-part series of 360 videos.
Voice-overs and on-screen text help build context through the translated Glasbrenner is the first American amputee to climb
voices of civilians and soldiers caught in the fray, but for the entire piece, Mount Everest.
viewers are stuck in a fixed point and, more important, are reliant on the PHOTO COURTESY OF SPORTS ILLUSTRATED /
journalist and a video producer to organize the story. LIFE VR
The AGE OF DYNAMIC STORYTELLING | 4

“Compared to a still photo assignment, 360 is the ideal medium for combat,”
said Maya Alleruzzo, the AP journalist who produced “House to House.” “The
stills were good, but couldn’t really capture the scene like the 360 images.”

This unique approach of expanding the traditionally fixed frame was an

effective entry point to show journalists and audiences the broader
possibilities of immersive media.

“360-video was a good introduction to train our journalists how to tell a story
in this medium, and it can lead us to approach the future of VR confident in
this unique method of storytelling,” explained Thomas Seymat, the head of
immersive storytelling at Euronews.

Reaching full immersion Immersive journalist Dan Archer (right) creates a 3-D
scan of Trevor, a subject of “Beneath These Restless
The next iteration of immersive media can be found in 3-D content driven by Skies,” an immersive experience on Section 8 housing
volumetric capture. This technology allows participants to freely move in Harlem, New York.
throughout a photo realistic scene and choose how they experience a story, PHOTO COURTESY OF HARRIET DEDMAN
forcing news professionals to rethink how they organize and construct stories.
The new approach promises the next revolution in publishing by immersing us
deep in a story.

“After the initial wow factor, the first thing people ask me when they get into a
360 story is, ‘What can I do?’” said Jesse Damiani, editor-at-large for the virtual
reality publication VRScout. “And if the answer is nothing, then the
marketability and viability of the experience drops exponentially; to drive
engagement, we need more tactile and tangible interactivity.”

For example, “Beneath These Restless Skies,” a story produced by the An early rendering of Trevor from “Beneath These
journalism studio Empathetic Media highlights the lack of change in Section Restless Skies,” a VR story created by Empathetic
8 housing in Harlem, New York. As participants choose their own path and Media that combines volumetric capture and CGI to
move through a 3-D scanned apartment, they are encouraged to pick up allow participants to walk through a Harlem, New York,
objects such as archival photographs to unlock pieces of the narrative and apartment.
understand the social climate of the location. PHOTO COURTESY OF EMPATHETIC MEDIA

“With ‘Beneath These Restless Skies,’ the challenge for us was how to chain
together these discrete pieces to form a narrative regardless of the order that a
user moves through them,” said Dan Archer, founder of Empathetic Media.

Note that when journalists give participants more freedom, they are also
trading off some of the narrative control.

“As a director of traditional content, you have the control of where the camera
is pointed, what is in view, when to transition, when to fade to black. We
relinquish much of this control to our viewers in a VR environment,” said Dan
Kempton an interactive editor at AP. Emblematic Group and Frontline’s “Greenland Melting”
combined the use of stereoscopic 360 video, 3-D
“Is this a bad thing? Maybe for some, but for those who are up to the challenge, scanning and CGI to allow participants to walk
discovering new techniques to solve issues that arise from your audience throughout a melting glacier.
running amok in your virtual story can be quite rewarding.” PHOTO COURTESY OF EMBLEMATIC GROUP
The AGE OF DYNAMIC STORYTELLING | 5

Emblematic Group, explored a similar approach with “Greenland Melting,” a

story told in partnership with Frontline and PBS. Using a tool called zBrush Storytelling models
the journalists were able to add a layer of depth to the 360 video captured on
Traditional
various icebergs. This added dimension allowed participants to move
throughout the glaciers, as 3-D models of NASA scientists explained the
effects of global warming on the environment.

Dynamic

Putting the viewer in control

Fundamentally, dynamic storytelling is the same as storytelling in its past and
current forms — journalists answer questions posed by their readers and
viewers. It’s just how those questions are being answered that’s changing.

A text Q&A, for example, is a very linear experience. Question 1 is followed by In dynamic storytelling, there is no predefined order
answer 1, followed by question 2, followed by answer 2, etc. to the story. News consumers can choose different
paths as they actively explore the experience.
In virtual reality, when answers are revealed depends on the participants,
who can choose when and which questions they want answered through
contextual 3-D objects, sound or on-screen text annotations.

“It’s all about putting the control in the user’s hands and bringing together as
many in-depth experiential pieces to them so they can navigate their way
through a story,” explained Brittany Peterson, an immersive journalist at
McClatchy.

Thinking through multiple perspectives

“When you include more interactivity by impacting someone’s senses as well
as giving them six degrees of freedom, it means there’s a stronger reaction, a
stronger commitment and ultimately a stronger memory of the experience,”
said Gabo Arora, founder of the United Nations’ VR unit.

One example of a story using this model is USA Today’s “Eisenhower VR,”
which allows a participant to move through the USS Eisenhower aircraft
carrier and experience the daily lives of crew members, ranging from the
captain to the cooks through a series of 3-D models, 360-degree videos,
animations and graphics.

“Non-linear storytelling is a more natural form of storytelling for this medium,”

said Niko Chauls, who worked as a director of emerging technology at USA
Today. “Much like how an infographic takes a complex set of information and
boils it down to its core elements to make it easier to understand, I think this
was an approach that accomplished this same effect and allowed users to
explore at their own pace.” Various scenes from USA Today’s “Eisenhower VR.”
This story combined a 3-D model of the USS
For Chris Papaleo, the executive director of emerging technologies at Hearst, Eisenhower with a series of 360-degree videos that a
one of the nation’s largest media conglomerates, non-linear storytelling participant would trigger as he or she moved through
represents a bigger shift than the move from print to digital or from desktop the 3-D model of the carrier.
to mobile. PHOTOS COURTESY OF USA TODAY
The AGE OF DYNAMIC STORYTELLING | 6

“It’s not as simple as transposing existing content to a new screen — it

requires us to learn new skills and use new tools,” explained Papaleo.

“The quality of the experience a viewer gets from VR is impossible to match,

VR allows a viewer to feel closer to the subject.”

Connecting technologies and platforms

This new model isn’t just about creating a single story. It instead benefits from
constructing story packages that can be experienced across multiple platforms
and, in some cases, with multiple technologies in play.

“Remaining innovative will involve pursuing the incredible and profound new
storytelling opportunities afforded by this rapidly changing technology
landscape,” said Graham Roberts, director of immersive platforms storytelling
at The New York Times. “It will also mean connecting the VR work we Empathetic Media partnered with the Washington Post
produce to other immersive technologies along a spectrum of storytelling.” to combine court documents, eye witness testimony
and 3-D imagery into this AR experience on the death
Beyond virtual reality, journalists are starting to explore augmented reality, of Freddie Grey in Baltimore. To access this story,
or AR, which entails projecting digital 3-D models onto a physical space using participants had to download the custom-made ARC
a mobile phone’s camera. application and scan a Washington Post logo.
PHOTO COURTESY OF EMPATHETIC MEDIA
The Washington Post, for example, partnered with Empathetic Media
to conceptualize an AR extension of the Post’s coverage on of the death
of Freddie Gray in Baltimore. Readers of the print edition could use
a smartphone app to scan the Post’s logo to uncover a 3-D recreation
of the event.

“Experimenting with new forms of immersive media is critical because news

organizations won’t be able to accurately predict which forms will be most
widely embraced by the news consuming public,”explained Jeremy Gilbert,
The Washington Post’s director of strategic initiatives.

“We have to try things and know what stories and techniques work best in any
form of immersive media so we’re ready when certain forms of media win out.”

Elsewhere in the industry, Sports Illustrated and LIFE VR’s “Capturing Everest”
used an augmented reality plug-in to connect their long-form text story to a
series of 360-degree videos that chronicled three climbers ascending up
Mount Everest.

“AR really closed the loop with what we were doing in print and what we were
doing with 360 content and allowed us to create a full-circle approach in
presenting a package with immersive media,” said Mia Tramz, managing editor LIFE VR used augmented reality to connect a long-
at LIFE VR, Time Inc.’s new VR unit. form text story text story on Jeff Glasbrenner, the first
American amputee to climb Mount Everest, to a series
“Context is such an important part of what journalists have to bring to the of 360-degree videos on the same subject. To access
table, and AR puts things in context through a real-world environment.” the videos, a participant scans the cover of the Sports
Illustrated issue with a mobile phone and the
In fact, AR may provide an entry point for more viewers because the 360-degree videos automatically start playing.
technology may reach consumers quicker than other existing platforms. PHOTO COURTESY OF LIFE VR
The AGE OF DYNAMIC STORYTELLING | 7

Carol Chainon, an entrepreneur and VR producer for the Global Editors

Network,the world’s largest association of news executives, explained that in
next few years we will start to see consumer products and features that will
allow for information to be gathered from the natural world through our
mobile devices. AR lays the groundwork for this to be possible.

“If we look at the spectrum of immersive technologies, AR and mixed reality

will likely be of most immediate use to a mass audience, because they will very
soon become ubiquitously available on our smartphones, thanks to Apple’s
ARKit, Google Tango and Facebook’s AR platform,” said Chainon.

The Apple, Google and Facebook platforms are new tools that enable a
streamlined development of augmented reality.

Rethinking audience participation

During the Summer of 2017, NBC News broadcasted a series of virtual NBC’s “Bill Nye in VR” event was one of the first
sit-downs with science icons like Bill Nye live on the social VR platform attempts for a media company to connect with its
AltSpace. Paul Cheung, director of digital news at NBC, said these talks have audience using social VR platforms like AltSpace.
challenged his newsroom to think about VR strategically and examine what it During the event, participants could ask questions
NBC’s experiment is a great example of how newsrooms can combine social comment with emojis in real time.
aspects found in communication platforms such as Skype, Google Hangouts PHOTO COURTESY OF NBC
and Facetime with the entertainment and educational value of a talk show or
the nightly newscast.

“Social VR can help us reimagine what it means to have a live audience,” said
Cheung. “In VR, you can reach a global audience and allow them to interact
with each other — you and your virtual environment can be very powerful.”

Early adopters might be skeptical of social VR due to the clunky avatars that
are commonly employed, but what’s most important is to reflect on the
potential for this format moving forward.

“There’s this uncanny moment that we see when people react, saying, ‘Oh, my
God, I’m part of the story,’” said Igal Nassima, founder of Superbright, a
creative studio in New York City that developed the technology for one of the
first reality shows in VR commissioned by Condé Nast.

“As storytellers and content creators, we need to continue to experiment with

that level of interaction to see where it can take us.”

In the future, participants will be able to replace rudimentary avatars with 3-D
scans of themselves, resulting in a more authentic feel to social VR.

“3-D scanning has greater implications behind the scenes, and I believe it will Condé Nast recently partnered with the New York City
become part of standardized broadcast techniques without audiences truly technology studio Superbright to create the first-ever
needing to understand what it is nor how it is being used,” said Kevin Joyce, dating show in virtual reality. Participants first had to
editor-in-chief of VR Focus, a publication covering the VR industry. be 3-D scanned to create their own virtual avatars
before meeting their blind date in VR.
The ability to truly interact with a global audience promises to reinvent the PHOTO COURTESY OF SUPERBRIGHT
The AGE OF DYNAMIC STORYTELLING | 8

current formats of talks shows, political debates and even town hall meetings “Just because the
storytelling format
by bringing the news to participants in virtual spaces.

is unique, it doesn’t
“The continuation of these technologies will be less focused on presence and
more focused on moving along a continuum of being present,” said Shawn

mean that we throw

Cheng, a New York City-based venture capitalist focused on immersive media
investments.

away all long-

Key challenges and concerns standing reporting
Despite the enthusiasm that surrounds these new technologies, there are
several challenges standing in the way of solidifying the dynamic storytelling
methods.”
approach across newsrooms. SCOTT MAYEROWITZ
BUSINESS EDITOR
ASSOCIATED PRESS
“A great story told well is a great story, no matter what the format. With VR,
lots of perennial questions have reemerged, as they do with all new frontiers
— for example, around journalism ethics and standards, about newsroom
workflows, technology, skills and user adoption,” explained Shazna Nessa, Tips for journalists exploring
global head of visuals at The Wall Street Journal. dynamic storytelling

Maintaining journalistic standards Put the audience at the center of the process.
User choice can drive higher levels of impact
As journalists experiment with these new technologies, they are also and participation. Stories can be structured
upholding certain editorial standards throughout the entire process. with the audience in mind to allow a
participant to uncover virtual elements while
“New technology brings wonderful opportunities for reinventing storytelling,” piecing together the narrative.
said Lisa Gibbs, AP’s director of news partnerships.
Create an experience that lives across
“But just because we have the capability to do something doesn’t mean we platforms. Including elements such as
should. Applying the same editorial, ethical standards to our innovations as interactive graphics, flat or 360 videos and
we would any traditional coverage initiative is critical to our journalistic photos that can distributed through traditional
mission.” channels can promote and drive more traffic to
the immersive story package as a whole.
Let’s say a few years down the road a reporter wants to volumetrically scan a
group of protestors marching down a crowded street. While he or she could Leverage the expertise of everyone in the
accurately capture the surrounding buildings and the street itself, the newsroom. Early adopters will become experts
individuals could become blurred and distorted. How should the news on this new storytelling model, but it doesn’t
professional handle this natural distortion from an ethical perspective? mean that knowledge should be confined to
only a small group. Share dynamic storytelling
“Journalists should present their editorial options, but should never make best practices with your peers and collaborate
assumptions,” said Scott Mayerowitz, an editor on AP’s business desk. “Just to expand editorial innovation.
because the storytelling format is unique, it doesn’t mean that we throw away
all long-standing reporting methods.” Test and iterate throughout the process.
Since dynamic storytelling doesn’t follow a
If a newsroom takes it upon itself to alter or recreate a scene without any prior structured formula, feedback is needed from
reference, it could be overstepping its bounds. One possible solution is to story conception, through design and
include a disclosure stating that distortions were left in place because the execution.
editorial team couldn’t determine the accuracy of a certain footage.
The AGE OF DYNAMIC STORYTELLING | 9

In industries like film and video game production, where 3-D modeling is “The technology
always moves faster
widely used, such distortions or inaccuracies are merely seen by the consumer
as poor production value that many people write-off because it’s a work of

than the ethics do.

fiction.

We have to ask
Journalism isn’t afforded that luxury. News organizations are starting to set
standards to uphold truth telling from the start, which is contributing to the

ourselves ‘Before we
progress immersive journalists have made thus far.

had VR, what were

“We’re at a stage now where neglecting ethics could be extremely
disruptive to our ability to do this kind of work,” warned Saleem Khan,

our standards for

founder of JoVRalism, a Toronto-based publication covering the intersection
of journalism and immersive media.

“This technology is still in its infancy, and if people decide they can’t trust truth telling? Now
that we have new
it, then what does it mean for the future? These doubts could create an
environment where people call everything they’re experiencing into question.”

Beyond the ethical implications in producing volumetric content, newsrooms technologies, do

those standards
are also paying attention to how they present it.

change? If so, why

For example, how should a reporter present the aftermath of a terrorist
attack or the chaos seen in a war zone? Inducing post-traumatic stress isn’t

and how can I

something for which a newsroom would want to be responsible.

explain that to the

“The more realistic that we make these immersive experiences the more we
have to consider what we are putting people into because the impact will be

public?’”
multiplied,” added Khan.

Including disclaimers before a story is a helpful strategy to warn participants

they are about to engage with disturbing environments. However, these AL THOMPKINS
SENIOR FACULTY MEMBER
shouldn’t be used as a sanitizer to protect a news organization. The question POYNTER INSTITUTE
to consider moving forward is how much is too much when dealing with
potentially sensitive images?

“Because VR impacts physiology in unique ways, immersive storytellers have to

be extra careful with their content,” alerted Sarah Hill, founder of the
immersive journalism studio StoryUp. “These challenges, when well-managed,
also present unique opportunities for storytellers to create impactful content
beyond just information and entertainment.”

Al Thompkins, a senior faculty member at Poynter, encouraged newsrooms

who are entering this field to set their own, non-negotiable standards before
seeking external technology partners.

“The technology always moves faster than the ethics do,” Thompkins observed.
“We have to ask ourselves ‘Before we had VR, what were our standards for A look at the equipment AP provides its immersive
truth telling? Now that we have new technologies, do those standards change? journalists shooting 360-degree video around
If so, why and how can I explain that to the public?’” the world.
AP PHOTO
The AGE OF DYNAMIC STORYTELLING | 10

New skills, tools and workflows

Dynamic storytelling requires new skills beyond traditional reporting. These
are capabilities more likely to be associated with graphic artists, game
developers and motion designers.

“VR reporters need to be fluent in 3-D animation, motion graphics, video

editing and technical rendering in order to produce stories in this emerging
medium,” observed Darrell Allen, an interactive journalism editor with AP.

Gaming programs such as Unity, Unreal and the 3-D modeling software Maya
LT provide a foundation for how stories using volumetric content are
produced. These programs enable media organizations to build 3-D
recreations of news events, to increase the interactivity level of their graphics
and to create AR extensions of text stories.

“For news publishers, the immersive nature offers a unique degree of focus
from its users. But news publishers are also challenged to provide an The 3-D scanning tool DepthKit combines the
experience that is interactive like a video game. That’s a difficult combination,” capabilities of DSLR camera with the Microsoft Kinect
said The Washington Post’s Gilbert. for depth sensing to create 3-D avatars of real people.
PHOTO COURTESY OF DEPTHKIT
James George, a founder of the volumetric scanning tool DepthKit, said this
emerging technology is becoming a new genre in of itself and requires skills
traditionally considered less important to news companies.

“The future of volumetric is to take the subject matter, themes and approach of
the current 360-video world and expand on that format to make it more
experiential, magical and explorable in a way that VR animation content is Developing 3-D models and virtual
right now,” George said. environments follows three steps
Cross-disciplinary collaboration is also important to the success of these 1. Conduct thorough journalistic research,
projects. AP has worked with a game developer and visual effects artist who find sources and visual references for
had worked, among other organizations, at the gaming giant Electronic Arts. the story.
Leveraging these skills, we produced a series of experiences generated through
computer graphics (CGI) that included an immersive look at the effects of 2. Build 3-D models of characters, buildings,
global warming and a virtual museum of the most dangerous invasive species in vehicles and trees that make up the news
the U.S. event. These elements can also be
volumetrically captured and processed
Another AP immersive experience using CGI examined the causes of through CGI.
Alzheimer’s disease from inside the brain and allowed participants to see
first-hand how the condition spreads. 3. Program the behaviors of these elements,
such as individuals’ facial expressions and
“The use of computer-generated imagery for “Alzheimer’s Disease: Exploring how people move through a virtual
the Brain” offers the participant a perspective they wouldn’t be able to see; environment. This entire process requires
in this case, it was a microscopic view inside the brain. CGI also gave us graphical knowledge and skills that were
the ability to better tell the story by allowing us to show events that would traditionally exclusively found in game
normally need expensive scientific equipment,” said Panagiotis Mouzakis, development.
an AP multimedia animation producer based in London.
The AGE OF DYNAMIC STORYTELLING | 11

Collaborating with individuals with non-traditional editorial skills can enable

newsrooms to accelerate their move into dynamic storytelling.

“Journalism is a fairly new entrant into the VR space, whereas game designers
and others have been embracing this new technology for years,” said Laura
Hertzfeld, program director at Journalism 360, a collaborative initiative by the
Knight Foundation, the Google News Lab and the Newseum to accelerate
immersive storytelling in journalism.

“Right now, the community experimenting in this new space is relatively small
— both within journalism and beyond — so people who are passionate about
the technology are eager to share their knowledge, and that can only be a good AP partnered with a game designer to producer
thing.” a series of CGI-driven stories, including this story
that highlights some of the United States’ most
While this kind of storytelling is still in its infancy, exploration of new invasive species.
technology is key. AP PHOTO

“There’s no how-to guide for immersive storytelling and I think there’s still a
sense of mystery around it — we’re trying to demystify it by testing different
camera setups and techniques, integrating 360 video players in our mobile
apps and sites, learning new development and design environments like Unity
and Apple’s ARKit, and charting new production workflows,” explained
Heart’s Chris Papaleo.

Al Jazeera’s Contrast VR unit is another example of knowledge sharing

throughout the newsroom. In addition to being a production unit for the Al
Jazeera network, Contrast also harbors the responsibility for training other
journalists around the world. This approach is giving journalists at Contrast, “Alzheimer’s Disease: Exploring the Brain” is an
like Zahra Rasool, the opportunity to beta-test some of the experiences her example of story that isn’t possible without the
unit is working on. capabilities of computer-generated imagery (CGI).
With the help of a game designer, journalists at AP
“Al Jazeera is a company that’s spread around the world and some of the were able to produce an unprecedented look at how
people that we work with haven’t been exposed to this kind of storytelling as this disease spreads throughout the brain.
compared to someone from the United States,” explained Rasool, who AP PHOTO
recently held a training session at the Al Jazeera bureau in Uganda.

“We hold regular training sessions for other journalists and take their feedback
to constantly iterate on our stories so that we can create the best stories
possible, even for those who have never experienced VR before.”

An important takeaway is to provide the space for dynamic storytelling

experimentation by building institutional expertise that can be shared
throughout the newsroom.

“The challenge for us, and for everyone right now, is creating the bridge
between what we can publish now through these quickly evolving, but still “Global Warming: Signs & Effects” simulates how
relatively nascent platforms and technologies, to what we will be able to do in the “greenhouse effect” works and highlights some
the future — a future that will probably arrive sooner than we expect,” associated climate trends and their influence on
acknowledged Roberts of The New York Times. global warming.
AP PHOTO
The AGE OF DYNAMIC STORYTELLING | 12

Navigating technology costs and constraints

One of the primary challenges in producing volumetric content centers
around the extensive production time and cost it takes to scan and combine
the people, places and objects that make up a 3-D environment. This new
content creation process results in copious amounts of data and large file
sizes, making it difficult to produce directly from the field.

To overcome the challenge of managing these gigantic data files, AP partnered

with AMD, a technology company that specializes in image processing and
graphics, to develop more realistic immersive media experiences. AMD
provided journalists at AP with powerful computers capable of significantly
increasing the speed of processing and producing 3-D content.
In 2015 AP partnered with AMD, a technology
company specializing in high-end image processing
Deniz Ergürel, head of the VR publication Haptical explained that volumetric
and graphics, to jumpstart their VR production.
capture technology enables newsrooms to serve their audience’s needs in an
PHOTO COURTESY OF AMD
entirely new fashion.

“The goal here isn’t to show off a new and shiny form of technology, but the
question newsrooms should be asking themselves is ‘what can this technology
bring to storytelling that others can’t?’” explained Ergürel.

Not all newsrooms will have the same level of resources and access to new
technology. However, media initiatives like Journalism 360 can provide
funding and share best practices. During the summer of 2017, Journalism 360
awarded $285,000 in grants to 11 projects around the world to encourage the
growth of immersive storytelling in journalism. Tips on how to experiment with
immersive media in the newsroom
“Experimentation is key to the future,” explained Erica Anderson of the Google
News Lab, a Journalism 360 partner. Create a small challenge fund and let teams
compete with proposals to experiment with
Beyond technical newsroom capabilities, access and user adoption present immersive projects that will spark new internal
itself as one of the final technical challenges for immersive journalism. thinking, whether or not they’re good enough
to make public.
“VR will inherently be more expensive as it moves into mainstream media,”
said Jonathan Albright, research director at the Tow Center for Digital Plan one-day bootcamps where participants
Journalism at Columbia University. start from scratch and end up with a group
project. They’ll get a taste for the state of the
“This will unfortunately create a digital ‘format divide’ for the people who can’t art and increased awareness of the medium’s
afford the expensive devices and high-bandwidth data plans required to access limitations and potential.
the content.”
Add highlight links to internal newsletters
noting VR experimentation done by other
news organizations to spread awareness
of the medium’s strengths and limitations.

Tips provided by Jeremy Caplan, director

of education at CUNY Graduate School
of Journalism.
The AGE OF DYNAMIC STORYTELLING | 13

As dynamic storytelling breaks

Dynamic vs. away from the traditional
traditional approach, here are some
storytelling important distinctions to keep
in mind.

Dynamic branched active participant

Traditional linear passive viewer

HOW IS THE STORY CONSTRUCTED?

Linear vs. branched story structure

Linear The order in which a story is presented is vital for it to be understood. More
important, linear storytelling requires there to be a “giver” of information — traditionally
journalists — and a “receiver” — the public.

Branched There is no predefined order to a story. Stories are created as interactive

experiences, where news consumers can choose different paths. Audiences actively explore
stories, rather than passively viewing them.

HOW WILL A PARTICIPANT CONSUME THIS EXPERIENCE?

Passive vs. active consumption

Passive Observers have limited control over how a story progresses. A journalist guides
them through the narrative and packages the information in a fixed order.

Active Participants have the ability to affect how their experience evolves. They have the
freedom to dictate how they consume information, whether that’s through movement,
interaction or story structure.

WHO IS THE AUDIENCE?

Observing vs. participating audience

Observer The producer of the story is in complete control, guiding an observant user in
specific ways to transmit certain information.

Participant Users command how they move through an environment and digest
information, with the freedom to participate, touch and interact with the people, places
and events that are presented.
The AGE OF DYNAMIC STORYTELLING | 14

Immersive media has many facets, and there are

Understanding many ways it can be implemented in a newsroom,
but just like any other technology, the more you
the technology know about a tool, the more effectively you can
use it.
involved
Monoscopic
36o video
Stereoscopic

Immersive content Augmented reality

3-D scan
Volumetric
CGI (computer-generated imagery)

360-degree video — Monoscopic The majority of 360-degree video is captured with a single
camera or camera rig. While the spherical image provides a sense of immersion, clarity
decreases at the edges of this sphere.

Example: AP’s “Trapped in Myanmar: Life inside a Rohingya camp”

360-degree video — Stereoscopic: A pair of 360-degree camera lens are placed side by side
to add depth between the foreground and background for a heightened sense of
immersion.

Example: NextVR’s coverage of the NBA Finals

Augmented / Mixed Reality 3-D models are projected onto physical surfaces using depth
sensors built into the cameras of mobile devices.

Example: Empathetic Media and The Washington Post coverage of Freddie Grey’s death

Volumetric — 3-D scan Real people, places and objects are scanned with depth sensors to
create elements that, when combined, form a 3-D environment.

Example: Emblematic Group’s “Out of Exile”

Volumetric — CGI (Computer-generated imagery) 3-D models and environments are

recreated based upon photo and video references through gaming software.

Many times, volumetric scans and CGI models are combined to create a more cohesive
presentation.

Example: AP’s “Alzheimer’s Disease: Exploring the Brain”

The AGE OF DYNAMIC STORYTELLING | 15

Scientific study on VR
user engagement
Measuring media immersion:
AP and Multimer study
Although news consumers may not consciously realize it, experiencing
immersive media activates thousands of physical and physiological stimuli.
These changes can result in participants becoming so engaged that they hardly
notice their surroundings.

To measure the psychological effects of immersive storytelling, we teamed up

with neuroscientists at Multimer, a New York City-based company born out of
the MIT Media Lab that has developed mobile biosensors used to measure a A participant fills out a survey on the Multimer mobile
person’s mental state as they performed different activities. app after watching a virtual story at AP’s VR study at
the NYC Media Lab on August 12, 2017. The
The study was conducted at New York University’s VR Lab MAGNET (Media participants were asked to explain what emotions were
and Game Network), a facility dedicated to studying the intersection of associated with each story and a specific moment in
emerging technology and culture. MAGNET’s Black Box Theater provided us each story that stood out to them.
with state-of-the-art equipment, including a motion-tracking system and AP PHOTO
room-scale VR headsets that allow participants to move through a virtual
scene.

The immersive experiences presented included three 360-degree video stories

as well as a CGI room-scale experience. These stories featured the fight
against ISIS in Iraq, a cultural parade in New Orleans, an exploration of the
ivory trade in Thailand and an underwater experience exploring biodiversity
and marine species.

Twelve study participants were paired with a Multimer staff member, who
helped them put on motion-capture suits, an EEG brainwave sensors and
heart-rate monitors. Staffers also calibrated each participant’s sensor readings
and monitored their movement through a motion-tracking system.

While study participants experienced each story in VR across multiple viewing

devices, our team recorded their brain activities, heart rates and body motions. Motion tracking sensors were placed on individuals
In addition, individuals filled out surveys in the Multimer data app asking how who participated in The Associated Press’ virtual
each story impacted them emotionally and whether there were particular reality study at the NYC Media Lab on August 12, 2017.
moments in each story that sparked their feelings. The motion tracking data showed that stories dynamic
in nature resulted in more movement by each
A detailed methodology for this study is outlined on Page 24 of this report. participant.
AP PHOTO
The AGE OF DYNAMIC STORYTELLING | 16

Measuring and interpreting physiological metrics

In order to understand the effects of VR on the brain, Multimer’s research
scientists first had to identify the core physiological metrics that can be
broken down, parsed and compared with the study participants’ qualitative
feedback.

These core metrics include two key “ingredients”: a participant’s overall

attention to a particular story, and their overall relaxation associated to that
same experience.

Quantifying levels of attention and relaxation involves examining the alpha

and theta brain waves that were collected throughout the study via Multimer’s
mobile app. In general, prior research indicates that brainwave alpha
frequency in the broad range of about 6.5–10.5 Hz is associated with a
significant level of attention. An increase in theta waves (4–7.75 Hz) is
associated with relaxation.
Arlene Ducao, founder of Multimer, and NYU research
These baseline readings provide a framework that can be further analyzed to assistant Alexis Trevizo observe participants of AP’s
uncover more specific states of mind using the contextual information of a virtual reality study at the NYC Media Lab on August
story, the device that an individual was using as well as the participant’s own 12, 2017. Ducao and Trevizo were responsible for
feedback. running the motion-capture system throughout
the study.
Multimer’s data scientists hypothesized the presence of key sentiments by AP PHOTO
looking for specific balances of the brain waves associated with attention and
relaxation, as well as analyzing them against heart rate and verbal feedback.
This was done using custom algorithms based on prior Multimer research and
the study participants’ verbal comments.

Note that these categories are meant to be suggestive rather than prescriptive
or fully descriptive; to be fully confirmed, they would require robust statistical
analysis and repeated data collection in the context of VR technology.

These sentiments include:

Open-mindedness Associated with an individual being attracted to a topic, but
not alarmed. For this metric, a participant exhibits less than 44 percent of the
overall range of attention, less than 53 percent of the overall range for Participants watch the AP’s “House to House: The
relaxation and an average of 77 heartbeats per minute. Battle for Mosul” on a VR cardboard as part of a virtual
reality study at the New York City Media Lab on
Fascination Associated with a relaxed interest in a topic. For this metric, a August 12, 2017. The participants were hooked up to
participant exhibits less than 57 percent of the overall range of attention, brainwave sensors, heart rate monitors and a motion
greater than 53 percent of the overall range for relaxation and an average of 82 tracking system in order to test how various VR stories
heartbeats per minute. affected their psychological state.
AP PHOTO
Stimulation Associated with an individual being more attentive than they are
relaxed. For this metric, a participant exhibits greater than 50 percent of the
overall range of attention, less than 53 percent of the overall range for
relaxation and an average of 82 heartbeats per minute.
The AGE OF DYNAMIC STORYTELLING | 17

Power / Intensity Associated with the lasting impact of the experience. For War zone immersive
reporting drives
this metric, a participant exhibits greater than 44 percent of the overall range
of attention, greater than 54 percent of the overall range for relaxation, and an

participant
average of 84 heartbeats per minute.

Open-mindedness
Heart rate

77 BPM
Attention

44%
Relaxation

53%
stimulation,
Fascination 82 BPM 57% 53% while science
Stimulation

Power / Intensity
82 bpm

84 bpm
50%

44%
50%

54%
and environment
stories build
Study findings open-mindedness.
The results allowed us to derive the following observations related to device
and story type.

Hands-free headsets drive higher levels of engagement.

Cardboard VR headset Room-scale VR
Open-mindedness 2 3 1
Fascination 2 3 1
Stimulation 1 2 3
Power / Intensity 2 1 3

Note: We ranked each device by sentiment on a scale of 1–3, with 1 being the highest.

We found participants had the highest levels of open-mindedness — being

attracted to a topic, but not alarmed — on the room-scale VR headset, while
the cardboard elicited the highest level of stimulation, which is associated
with individuals being more attentive than they are relaxed. Based on
participants’ comments, this may be connected to the higher levels of image
quality on a room-scale experience and the ability to freely move around the
room.

The high levels of stimulation for the cardboard device may sound
counterintuitive, but this heightened sensory reading is associated with low
image quality and the struggle to comprehend a blurred image. In addition,
the requirement for cardboard users to hold the device to their face also could
contribute to this result.

The VR headsets, which only enable a participant to watch a story from a fixed A study participant watches the Associated Press’
space, as opposed to move through the scene, elicited the highest overall “Thai Elephants: Surviving the 21st Century” on the
power / intensity. This is associated with the lasting impact of the experience Samsung Gear VR headset while wearing Multimer’s
and may be a result of combining relatively intense stories with a more biosensor. The biosensor clips onto an individual’s
“immersive” device. earlobe while another sensor rests on their forehead.
AP PHOTO
The AGE OF DYNAMIC STORYTELLING | 18

War zone immersive reporting drives participant stimulation, while science

and environment stories build open-mindedness.
Conflict Culture and Beat Science and
“House to lifestyle reporting environment
House” “The Second “Thai “The Blu”
Line” Elephants”

Open-Mindedness 4 3 2 1
Fascination 4 1 2 3
Stimulation 1 3 2 4
Power / Intensity 1 2 3 4

Note: We ranked each experience by sentiment on a scale of 1–4, with 1 again being
the highest. A study participant watches “The Blu,” a room-scale
virtual reality story, on the HTC Vive during AP’s VR
All physiological metrics were lowest for “The Blu,” an experience exploring study at the NYC Media Lab on August 12, 2017.
marine biodiversity, which was the single experience tested on a room-scale AP PHOTO
level. This may suggest a higher level of engaged open-mindedness (being
attracted by a topic). However, more scientific testing is needed to determine
if this sentiment is caused by the story, the device or another factor entirely.

Metrics associated with stimulation and power / intensity — related to the

lasting impact of the experience — were highest for “The battle for Mosul,” a
story documenting the fight against ISIS in a major city in northern Iraq.
Across those metrics, this particular experiences measured 4 to 6 percent
higher than other experience. The scenes shot in the Iraqi city of Mosul also
elicited the most participant comments related to strife and fear.

Metrics associated with fascination — eliciting higher levels of attentiveness

than relaxation — were highest for “The Second Line,” a story that celebrates
multiculturalism in New Orleans. Those metrics were 5 to 7 percent higher
than other stories. It also garnered the most comments describing the
experience as “pleasant.”

The more dynamic an experience,

the higher the engagement
The initial results of our study surfaced that immersive experiences, allowing
participants to freely move through a virtual environment, drove the highest
levels of open-mindedness and fascination. These findings demonstrate that
while study participants were playing close attention to the content, they did
A study participant watches the Associated Press’
so in a state of relaxation — speaking to the overall sense of being present and
“The Second Line: A parade against violence” on the
immersed in the scene. For story topics specifically, our study also found that
Samsung Gear VR headset while being connected to
VR experiences related to science and discovery, presented in a room-scale
Multimer’s biosensor and a motion tracking system.
environment, drove the highest levels of open-mindedness and fascination.
AP PHOTO

On the other hand, the analysis revealed that experiences in a VR headset

resulted in heightened levels of stimulation and intensity. Stimulation and
intensity both involve an elevated heart rate and levels of attention.
The AGE OF DYNAMIC STORYTELLING | 19

We noted that presenting war and conflict stories such as “House to House:
The battle for Mosul” in a VR headset can drive higher levels of engagement Takeaways to retain
when compared to the cardboard.
Journalists working in this area of immersive
Experiences such as “The Second Line: A parade against violence,” featuring media might drive the following conclusions.
culture and lifestyle topics, aligned most closely with a sense of fascination.
Based on our results showing that room-scale VR had the highest level of –– While the cardboard enabled the entry
fascination, this content could be optimized if distributed in a room-scale point for immersive media experiences,
environment. our data showed there was a significant
struggle viewing stories with that device.
Extrapolating how audiences implicitly react to different forms of stories
across media environments can help newsrooms meet consumers’ needs and –– Room-scale VR drove the highest level
improve upon the overall impact of their reporting. of engagement, allowing participants to
not only pay attention to the experience,
“Creating a new media type requires storytellers to suspend some of their own but they did so in a state of relaxation.
preconceived notions or biases,” said Ken Romano, a product director with AP.
–– Stories that cover war zones, riots
“Immersive experiences are ultimately being built for their users. Involving and other forms of conflict drive higher
them in the process and monitoring their behavior is going to uncover engagement when consumed in VR
incredibly valuable information that can be used to refine future story headsets as opposed to the cardboard.
approaches.”
–– News topics covering science, nature
and exploration benefit from room-scale
VR presentation.
The AGE OF DYNAMIC STORYTELLING | 20

Conclusions
It’s difficult to fathom that just two decades ago the internet barely existed Highly immersive
stories will be
and print media was still as central to society as it was 100 years before.

gathered using the

“We take for granted the immediacy and rich visuals afforded to us by sharing
stories through pocket-sized rectangles of glass we seemingly can’t live

same journalistic
without,” said The New York Times’ Roberts. “But as amazing as the mobile
revolution has been, the current method of storytelling, distribution and

methods of
consumption shouldn’t be viewed as the final form factor for storytelling.”

contextualization
Much sooner than we may realize, 3-D content will become commonplace. As
the technology continues to advance and become more accessible, the current

and verification, but

notion of reporting may evolve into something entirely new.

will be presented in
“Companies like Intel and Qualcomm are making technology that allows us to
capture objects and process them into 3-D models very quickly with our cell

a manner that’s
phones,” said Emblematic Group’s Nonny De La Peña. “This will continue to
challenge journalists on how they want to capture the world.”

“The push for the future will be to continue to break that fixed frame by explorable.
thinking about how our body is placed in a space and what the space is like.”

Immersive media will bring stories to life

360-degree video allowed journalists to expand their frame of view by making
stories experiential. Volumetric capture and 3-D content will likely take this
approach to an entirely new level.

Highly immersive stories will be gathered using the same journalistic methods
of contextualization and verification, but will be presented in a manner that’s
explorable.

“One can easily imagine smells being added into virtual reality experiences
along with the wind or the feel of a hot sun,” said the AP’s Scott Mayerowitz.
“Whatever it is, the approach will remain the same — bringing people an
unbiased approach to newsgathering and taking them to places they would
otherwise not be able to visit.”

Journalists and consumers will have to relearn

how to produce and consume 3-D content
Volumetric broadcasting is vulnerable to the errors found in traditional media.
AP Multimedia Video Graphics Manager Darrell Allen
In fact, the risk of creating a biased report increases when a newsroom is able (left) and former AP interactives editor Nathan
to fully control, model and replicate an environment. It’s up to journalists to Griffiths use a Matterport camera to 3-D scan the
set editorial standards and ethics as more of this technology enters the penthouse suite at the Four Seasons in New York
mainstream of the news business. for “The Suite Life,” one of the AP’s first stories told
in virtual reality.
AP PHOTO
The AGE OF DYNAMIC STORYTELLING | 21

“One of the biggest challenges newsrooms face is handling the uncertainty “Technology
enhances the
associated with investments in VR equipment, training, production and
distribution,” said Jeremy Caplan, the director of education at the Tow-Knight

storytelling,
Center for Entrepreneurial Journalism. “Being a first mover is costly and risky
when standards, tools and platforms are evolving so quickly.”

There is still a great learning curve for the audience, too. Most participants but at the end
of the day, the
continue to view dynamic storytelling projects on smartphones or desktop
computers. Training them on how to interact with the stories in virtual reality

story is what
will take time and a concerted effort from news organizations.

Feasibility, profitability and distribution still remain a challenge matters most.”

As with other nascent technologies, there are significant hurdles to staff
training, costs and profitability of the final products. ZAHRA RASOOL
JOURNALIST
CONTRAST VR, AL JAZEERA
Producing immersive media requires specialized training for the journalists,
but it also takes longer explaining to sources what the requirements and the
vision for the project are.

Despite the rapid growth in VR headset sales, immersive media consumers are
still a small percentage of overall viewers. Distribution of immersive content
remains a challenge, and many small news organizations are not set up to
distribute this type of content.

Much of the initial funding is coming from device manufacturers and

platforms wishing to drive user consumption. As our study suggests,
participants’ attention levels are higher with immersive stories. That type of
engagement will ultimately make these formats more attractive to advertisers.
As 3-D content becomes prevalent, we can expect new monetization
opportunities to emerge.

Immersive media relies on building a culture of experimentation Associated Press staffers, from left, Michael Rubin,
In the short term, the cost of implementation may be a barrier to entry for Bernat Armangue, Dennis Passa, Tassanee Vejponsa
many news organizations. But as the technology matures and user adoption and Koji Ueda discuss their team project during AP’s
rises, journalists will rely on experimentation in order to develop best Virtual Reality Boot Camp in Bangkok
practices. AP PHOTO / CHARLES DHARAPAK

“The insights gained from testing are extremely valuable, but the economics
of early-phase innovation requires that we be focused and disciplined when it
comes to product development and creative when it comes to partnerships
and deal-making,” said Hearst’s Papaleo.

At its core, dynamic storytelling challenges us to rethink how we conceive,

construct and distribute stories to the public. The technology gives us a wider
range of possibilities, but also raises new questions.

“Technology enhances the storytelling, but at the end of the day, the story is
what matters most,” said Contrast VR’s Rasool.
The AGE OF DYNAMIC STORYTELLING | 22

About the authors

Francesco Marconi is the manager of strategic planning and development at
The Associated Press. He is an affiliate researcher at the MIT Media Lab and
an innovation fellow at The Tow Center for Digital Journalism at Columbia
University. Follow him on Twitter at @fpmarconi.

Taylor Nakagawa is an immersive media fellow at The Associated Press and a

master’s candidate at the Missouri School of Journalism. Taylor is a recipient
of the Scripps Howard Foundation Scholarship. Follow him on Twitter at
@tknakagawa.

The following Associated Press staff contributed to this report

Darrell Allen, multimedia and video graphics manager
Maya Alleruzzo, chief photo editor for the Middle East
Jaime Holguin, manager of news development
Dan Kempton, interactive editor
Jim Kennedy, senior vice president of strategy and enterprise development
Jake Kreinberg, marketing manager
Scott Mayorwitz, business news editor
Vincent Trippodi, vice president of R&D
Panagiotis Mouzakis, multimedia animation producer
Ken Romano, product director
Lisa Gibbs, director of news partnerships

Thank you to these experts who shared their insights

Thomas Seymat (Euronews)
Ole Krogsgaard (Euronews)
Jesse Damiani (VRScout)
James George (DepthKit)
Gabo Arora (United Nations VR)
Zahra Rasool (Contrast VR, Al Jazeera)
Michael Mandary (Tulane University)
Molly DeWolf Swenson (RYOT)
Niko Chauls (formally USA Today)
Brittany Peterson (McClatchy)
Igal Nassima (Superbright)
Alexy Furman (Aftermath VR)
Al Tompkins (Poynter)
Indira Lakshmanan (Poynter)
Mia Tramz (LIFE VR, Time Inc.)
Paul Cheung (NBC News)
Eric Shamlin (Secret Location)
Dan Archer (Empathetic Media)
Shawn Cheng (Venture Capitalist)
Justin Hendrix (NYC Media Lab)
Sarah Hill (StoryUp VR)
Chris Papaleo (Hearst)
Graham Roberts (The New York Times)
Carol Chainon (Global Editors Network)
The AGE OF DYNAMIC STORYTELLING | 23

Laura Hertzfeld (Journalism 360)

Nonny De La Peña (Emblematic Group)
Deniz Ergürel (Haptical)
Sasa Marinkovic (AMD)
Saleem Khan (JoVRnalism)
Jeremy Caplan ( CUNY Journalism School)
Bob Sacha (CUNY Journalism School)
Jeremy Gilbert (The Washington Post)
Emilio Garcia-Ruiz (The Washington Post)
Kevin Joyce (VR Focus)
Jonathan Albright (Tow Center at Columbia University)
Erica Anderson (Google News Lab)
Shazna Nessa (Wall Street Journal)

And a special thanks to the Multimer research team

Arlene Ducao
Ilias Koen
Yapah Berry
Phoebe Zhang
Tommy Mitchell
Tania van Bergen
Amiri Haynes

Multimer is an New York City-based MIT spinoff, that offers a unique,

quantitative way to build empathy by measuring a population’s sentiment and
experience. Its products include MindRider, a location-aware biosensor
system, and a data analytics platform that integrates biosensor and geographic
data. We are a proud member of [Link] and an awardee of the National
Science Foundation SBIR program.
The AGE OF DYNAMIC STORYTELLING | 24

Works cited in AP and Multimer study

Egner, Tobias, and John H. Gruzelier. “EEG biofeedback of low beta band
components: frequency-specific effects on variables of attention and event-
related brain potentials.” Clinical neurophysiology 115.1 (2004): 131–139.

Jacobs, Gregg D., and Richard Friedman. “EEG spectral analysis of relaxation
techniques.” Applied psychophysiology and biofeedback 29.4 (2004): 245–254.

Klimesch, Wolfgang, et al. “Induced alpha band power changes in the human
EEG and attention.” Neuroscience letters 244.2 (1998): 73–76.

Teplan, Michal. “Fundamentals of EEG measurement.” Measurement science

review 2.2 (2002): 1–11.

Williams, John D., and John H. Gruzelier. “Differentiation of hypnosis and

relaxation by analysis of narrow band theta and alpha frequencies.”
International Journal of Clinical and Experimental Hypnosis 49.3 (2001):
185–206.

AP and Multimer research

methodology
This study measured heart-rate (beats per minute), EEG brainwaves (alpha,
beta, delta, gamma, theta frequency bands) from a single electrode device, and
recorded participant comments. The sample size was 12, and simple
percentages were used to compare outcomes; further statistical analysis is
required to determine correlation and statistical significance. The process of
this study followed these steps:

12 participants were scheduled over three 2.5-hour sessions. There were four
participants in each session. Each participant was compensated with an AP gift
bag, snacks, and refreshments.

When participants arrived, they first signed waiver forms acknowledging they
understood how data would be recorded.

Participants were paired with a Multimer staff member, who helped

participants to outfit themselves in a motion-capture suit, an EEG sensor, and
a heart-rate monitor. The Multimer staff member also helped to calibrate each
participant’s sensor readings and monitor their movement through a motion-
tracking system.

An AP staff member paired participants with VR technologies in this manner

Over the course of the session, AP staff members helped participants to rotate
between three stations, with the following VR technologies and stories:
The AGE OF DYNAMIC STORYTELLING | 25

Cardboard (New Orleans, Mosul, Elephants)

VR headset (New Orleans, Mosul, Elephants)

Room-scale VR (The Blu)

After a participant finished watching each story, a Multimer staff member

facilitated the participant to fill out a short survey outlining the emotions
sparked by it and a specific moment in the story that stood out to their.

The three datasets (EEG, heart rate and motion) were then cleaned, parsed,
and analyzed using several Python analysis libraries. Machine learning was not
used for this initial analysis, but it can be used as part of a deeper analysis with
this data.

Relevant research and academic

resources
MIT Media Lab is an interdisciplinary research laboratory at the Massachusetts
Institute of Technology that devotes itself to projects at the convergence of
technology, multimedia, sciences, art and design, as well as to groups such as
the Laboratory of Social Machines and Civic Media that work to develop
technologies and solutions for the journalism community.

Nieman Foundation at Harvard University works to promote and elevate the

standards of journalism, while it’s affiliate, Nieman Lab, publishes insights and
industry updates on the latest innovations in the news media industry.

NYC Media Lab connects digital media and technology companies with
universities and research centers to explore and develop new technologies,
including artificial intelligence.

Reynolds Journalism Institute is a center affiliated with the Missouri School

of Journalism that focuses on researching and testing new models of
journalism in this era of technological advances.

Tow Center for Digital Journalism is a research center at the Columbia

Journalism School that explores the intersection between technology and
journalism, focusing particularly on the ways technology can improve
journalistic practices and consumption.

NYU MAGNET NYU’s media and games network is a center for the faculty and
students across the NYU network who are interested in the intersection of
culture and technology.
The AGE OF DYNAMIC STORYTELLING | 26

Stay informed
AP Insights introduces readers to emerging technologies while presenting
industry insights regarding their implications.

Columbia Journalism Review is a bimonthly publication of the Columbia

University Graduate School of Journalism that explores news and media
industry trends, often as they relate to new technologies.

[Link] is a website that covers the intersection of technology and

journalism.

MediaShift provides industry insights regarding journalism-based

technologies.

Nieman Lab publishes insights and industry updates on the latest innovations
in the news media industry.

The NYC Media Lab newsletter is a weekly digital newsletter that details the
most recent advances in artificial intelligence, virtual reality, augmented
reality and data science.

Storybench covers the world of digital storytelling and media innovation.

The Tow Center blog explores the most recent information regarding the
intersection of journalism and technology.

Journalism 360 is a collaboration among the Google New Lab, the Newseum
and the Online News Association dedicated to accelerating immersive
storytelling in news.
The definitive source
for journalism innovation
Working on projects related to dynamic storytelling?
Get in touch with us: fmarconi@[Link]