It’s June 2017, and we are just over three years into the consumer immersive storytelling revolution that was kickstarted by Facebook’s purchase of the virtual reality headset maker Oculus Rift for nearly $3bn (more than the $2bn first reported). The industry faces big challenges (of public awareness about the technology and how to find and consume it as part of daily life), while at the same time offering great opportunities to content makers everywhere.

The challenges

Multiple formats: There are many and fragmented formats of immersive storytelling – whether it is in virtual reality (VR), augmented reality (AR), mixed reality (MR); wherever content lies on the immersive storytelling spectrum. The picture is clear (if fast changing) to the tech world but not too many people beyond it.

Fragmented distribution: VR and AR can be consumed on laptops, mobile phones, mobile headsets with tracking capability (Google Daydream of Samsung Gear) or high-powered headsets linked to a laptop (Oculus Rift, HTC Vive). The choice of hardware is enormous and growing, with some of it coming at a high price. Chip technology and screen-quality advances will change the picture in months and years to come. But for now, mobile gives reach and the high-end headsets give ultimate quality.

The challenge in finding content: This fragmentation sets a great challenge for people looking for the best content. YouTube and Facebook have given 360-degree video massive reach. The high-end headsets have their own content stores. But how do users find the best content? The content that sets standards.

And how do we cross the awareness threshold? With all the above there remains (for all the talk of a hot sector, and exciting news announcements daily) a need for the industry to bring consumers, and publishers, across what the super-thoughtful marketing writer Seth Godin calls the awareness threshold. The need to get consumers to challenge a received opinion (in this case that VR might just be for sun-deprived gaming geeks and developers) and to actually try the latest product. And thus discover what it is, as the first step to becoming a regular consumer. That’s the plan.

The opportunities

Cheap hardware: Consumer 360 cameras are here, and they work. A moment of liberation for production and post-production where once makers fiddled with multiple memory cards and over-heating batteries, followed by hours of semi-manual synchronising of multiple video feeds. Those days are in effect over.

Cheap software: There are now free or cheap 360-degree plugins for all the standard video editing software suites.

Knowledge hubs: Journalism 360 was set up to share information provided by pioneers in immersive storytelling for the benefit of all. It now has backing from Google News Lab, Knight Foundation and Online News Association and just ran its first innovation awards open to all immersive storytellers. (Declaration of interest: I am an ambassador for Journalism 360.)

VR blogs and VR podcasts: There are some outstanding blogs and podcasts, showcasing breaking news and the best recent productions. Here are just four that are particularly worth catching:

And then there is the challenge that is also an opportunity. The need for a lot of good content, and for storytelling talent from other media – film, photojournalism, mobile journalism, theatre – to come into immersive storytelling to meet that need. The picture might be big and fairly confusing, but there is a lot going on, plenty of room for creative energy, and a big demand for high-quality work.

How to get started in immersive storytelling? And make a difference, too

I would suggest that one of the immediate opportunities in content making is to stay up close and personal in virtual storytelling (whether in 360-degree video or computer-generated narratives). To think one-on-one and to experiment with telling immersive stories with a sense of close proximity, and an awareness of the space around the viewer. Those experiments, those learnings, will remain valuable when the market works out where and how the mass market engages with immersive storytelling.

By focusing on the undoubted opportunities, we can make immersive storytelling a format that can transform our lives in multiple ways as:

  • Consumers of content
  • Members of society
  • Consumers of news coverage and other immersive content
  • Subjects of that immersive content.

At a time when the news industry in particular needs what the journalistic visionary Mark Little has called “trust architects” – to re-establish a healthy relationship between democratic government, news publishing, and news consumption – immersive storytelling (with its power to put viewers at the heart of the story, and to give them a special agency, an element of choice) has an important part to play.

The importance of doing something sustainable

In looking at the opportunities – what is cheap, easy and repeatable within a self-sustaining business model – we have recently been focusing at immersiv.ly (a company I founded in 2014 to make news content in virtual reality) on experiment with one-to-one 360-degree video. Where you the viewer are addressed directly by the subject of the video, whether that subject is a poet, an entrepreneur, a cook, a singer an artist or a historian. It’s a close-field approach (where your subject is between 1m and 1.50m from the camera) which has worked well for a number of pioneers in VR. With the advent of consumer cameras and free VR plugins to industry standard editing software, it is an ideal way for content-making talent to take their first experiments in what is still a nascent format. It’s far from being too late to join the party!

And the question of scale is important, as the industry needs to make it easy and affordable for makers from other media to transition into immersive storytelling. To start small and to experiment. And for this it is important to avoid limited thinking on what makes a good subject for storytelling in virtual reality or 360-degree video. And to be wary of the push for pixel-perfect complexity (and resulting cost to makers and users) which stands as a barrier for many people looking to enter the immersive storytelling field.

The power of close field in 360-degree video

In the past three years, there has been much discussion about what content, what subjects, whether news or otherwise, work well in virtual reality.

One of the most useful answers is “content that takes the viewer to somewhere they could not otherwise go”. To a truly virtual world, created digitally (whether in video or computer generated narrative).

But it is important to keep an open mind when defining what that “somewhere” is. In my experience that somewhere is either spectacular, and panoramic, or intimate, and close field.

And that the close field can be enormously powerful.

A lot of the early-stage news documentaries in 360 video, from 2014-15, were related to news stories of global importance or made to convey the needs of overseas charities and non-profit organisations.

  • The Polar Sea by arte.tv. Explores the drastically changing Arctic landscape as seen through the eyes of the Inuit.
  • Harvest of Change from the Des Moines Register. A multimedia narrative of life on an Iowa farm, opening with helicopter footage over the open plains.
  • Clouds over Sidra, by Gabo Arora of the United Nations, and Chris Milk and his VRSE team (now rebranded Within). The story of a young girl and her family’s life in a Syrian refugee camp.
  • The Million Man March, for police accountability in New York, by Spike Jonze and Chris Milk for Vice News.
  • Hong Kong Unrest. Filmed by Edward Miller for immersiv.ly in September 2014.

The common factor about these pieces, published in 2014-15, seems, at first glance, to be about scale. The panoramic view of big events, big topics or big spaces – oceans, plazas, deserts and the skyscrapered streets of Hong Kong or New York. But looking back now, an interesting thing, taking these stories together, is the power of intimate scale, of incidental detail seen through daily life. Sometimes it is in plain sight. Sometimes it is on the opposite side of the spherical video, away from the “big” action, waiting to be discovered.

  • Harvest of Change is about the dynamics of a family farm. The daily challenges.
  • The Million March is a case study in the close-quarters challenge of a big demonstration.
  • The compelling discovery in editing Hong Kong Unrest was the incidental discovery of everyday life (shopping, catching buses) that continued even as police faced off against large demonstrations, yards away (on the opposite side of the 360 sphere).
  • Clouds over Sidra has garnered headlines for the success that the UN has shown from fund-raising associated with showing it at special events or as part of fundraising on the streets of big cities. The Sidra in question is not a place, but a young girl. And the power of the piece is in being an engaged witness of her family’s life, acquiring incidental information from the viewer’s 360-degree encounter with their harsh existence in a refugee camp.

The power of Clouds over Sidra – Chris Milk memorably associated this piece with his vision of virtual reality as an “empathy machine” – is in part the experience of being placed at the heart of family life, translated to an unforgiving environment by world events. And in part it is down to Gabo Arora’s integrity as a journalist, winning the trust of his subjects in a refugee camp by spending time with them, at a time when much refugee coverage was the product of brief press events or news conferences. He describes the process in a fascinating interview with Kent Bye in his Voices of VR podcast.

That intimate, close field, explored by Arora in Clouds over Sidra, has gone on to host some of the most powerful VR experiences in the past 3 years, whether in 360-degree video or fully immersive computer-generated narratives. And these models are important now because the advent of consumer cameras and consumer binaural sound recorders means it is a format easily envisioned and captured by solo film-makers and guerrilla story-tellers. Those who need to travel light and film and edit against the clock.

What the standard-setting film-makers in 360 say

Some of the most beautiful 360-degree video work of the past two years has been done by the Felix and Paul studio, derived from their cutting-edge custom camera technology (where they have pushed the envelope on the effective use of stereoscopic video) and their artistic vision and finesse. They had some interesting things to say about close-field 360 video at the SXSW conference in Austin, Texas, in 2017. And here I am indebted to the reporting on Twitter of my friend Robert Hernandez of the University of Southern California. Professor Hernandez tweeted notes from the Felix and Paul master class:

Near field is most important part of a #VRshot for presence. Composing depth is more important than 360. @felixandpaul

The key terms here are “near field”, “presence”, “composing depth” and “360”.

Presence is the most sought-after quality in any virtual reality experience. Giving the viewer the sense that they are “present” in another world.

And it is valuable that Felix and Paul, whose technical prowess allows them be “pixel perfect” at the grandest scale, should so value “near field” (within three to six feet of the camera) when they are composing their shots. They highlight an important principle for anyone looking to storyboard a 360 or VR piece: to envision first what is happening in the close field and then to document how that relates to what is happening in the outer fields of the 360 camera’s field of view. Whether it is lights, props or action. The film-maker is composing in spheres.

Felix and Paul also noted that “composing depth is more important than 360”. And, for me, that “more important than 360” counters a particular limiting belief that all 360 experiences “must” somehow be fully alive, fully populated with prompts, graphics and incident at all times in every nook and cranny. Or that they somehow lose validity.

It’s all a question of degree of what is and isn’t important: there is a clear focus (a main subject) that a 360-degree film maker can have in mind in any spherical shot. If the content of the remainder of the sphere adds context, information or visual richness, so much the better. There’s great value for viewer interest and viewer retention in the space.

But the received opinion (and it is out there) that the viewer must somehow be “compelled” to look around a 360 space for it to be valid – rather than merely having an interesting and fulfilling opportunity to do so – that they must physically turn their neck in a full circle, is a case of distorted priorities. That received opinion, that invented sense of compulsion, can lead to a point where an over-populated sphere becomes a distracting carnival rather than an immersive space. Where even a convinced consumer of 360 video cries “less is more”.

The simple message of big budget pieces

Two noteworthy big-budget pieces of 2016-17 both show the power of the performative gaze in 360-degree video, used in a straightforward way. Each has a simple editorial purpose. One that is clear to the viewer.

1) New York Times noir: The New York Times’s LA Noir season in 360-degree video, produced in late 2016, and featuring some of Hollywood’s biggest stars, was shot on a specially built set, where the exaggerated lighting design of that genre play particularly well in 360 shots and lead-ins. The stories are cleverly related in tone and concision. It’s a big budget piece, beautifully scripted and executed by serious talent.

But the most memorable thing is the straightforward approach to shooting, and (importantly) the simple approach to explaining the experience to the audience.

You the viewer (who are represented by the 360 camera) are told (but unobtrusively) that you are the barman listening to an agitated, fugitive played by Don Cheadle; or you are the estranged husband on an agitated character played by Natalie Portman. You are assigned a role.

And in both case the star performers look straight down the lens. They are addressing you. You are with them in the shot. It’s compelling and intriguing. And, fundamentally, a very simple storytelling approach. It works. You are prompted and you believe.

2) Hallelujah: The Lytro light-field camera is one of the technical wonders of the video world. Capturing multiple shots and composing them into a physical matrix where the viewer can get a sense of moving around within that volume. In their showpiece film of a concert, Hallelujah (made with Chris Milk’s VR studio, Within), the power is in your direct address by the solo singer, part of a church choir. And of the intimate sense of presence. The tech fidelity is mind-blowing but the example of the power of one-to-one address is a simple, graspable one.

The editorial lessons to be learnt from the high-fidelity LA Noir or Hallelujah are just as valuable to someone wielding a mobile or lightweight consumer 360-degree camera and a lavaliere mic.

You the camera: The challenge of acting ‘around’ rather than ‘to’ the audience

At immersiv.ly our first experiments with having characters work “to”, or rather “around”, the camera involved a performance of Shakespeare’s Twelfth Night. We filmed the passage, which had been prepared by two of the actors – Richard Leeming (who plays Malvolio) and Alex Hope (who plays Andrew Aguecheek) – in Mount Pleasant Studio (green-screened for the occasion) in London in September 2014. The performance was directed by Edward Miller and the stitched video then green-screened into a spherical photo of a garden with a suitably Shakespearean backdrop. It is the famous gulling scene, where Malvolio is tricked by a forged letter into imagining that Olivia, the object of his hopeless admiration, in fact, loves him, and where those who mock his being “gulled” do so by talking “over” his soliloquy while concealed behind a nearby hedge.
We shot the performance in three repeat takes with the camera (representing you, the on-stage witness) in three different positions (to examine the varied effect the different positions had):
  • Position 1: You are at the heart of the performance, with the actors all around you, but treating you the camera as a “ghost” (the actors are not working to the camera).
  • Position 2: You are a third party, watching the scene at a close remove. You are again treated as a ghost.
  • Position 3: You are a third party, but close to the protagonist, Malvolio. A close party to his unwitting embarrassment. There is a sense of his engaging with you, the camera.
Our first takeaway was that putting the camera at the very heart of the action (Position 1) created a physical challenge to the viewer, who sometimes has to spin left and right to pick up on the audio cues around them (we placed the actors’ individual audio tracks in the sphere, using Unity game engine). Position 2, with you the third party at a remove, felt like a natural viewing position, but somewhat distant from the heart of the matter. Position 3 (where the viewer is close to Malvolio) gives the greatest sense of “presence” with his plight.
You can watch the published scene, where the editing took parts from each of the three camera positions, on Shakespeare 360, an immersiv.ly app in the App Store.
Our next discovery of how powerfully a subject can work with a 360-degree camera came with a later project: the filming of our first travel documentary series, at the home of cinema.

The journalistic power of the auto-portrait

In 2015, immersiv.ly was lucky enough to collaborate with Jean Yves Chainon and Carole Chainon of JYC VR in co-producing our first series of travel documentaries, set in Los Angeles. At first, we played with working with an interviewer or with two “characters” talking to each other. But in part by default or by the contingency of filming on the LA streets we found ourselves increasingly asking our subjects – art gallery directors, fitness models, social entrepreneurs, restaurant managers, beachside flaneurs – to tell their story straight to camera. Straight to the 360 rig. And the results we discovered in post-production process (in the days before live previews on 360 rigs) were compelling. We found that we had captured a powerful patchwork of personal testimonies, of auto-portraits, where the subject takes the viewer into their confidence. That is the power of looking straight into a 360 array or lens. We had a 360 version of the city patchwork narrative approach so beautifully essayed by Brandon Stanton at Humans of New York.

This discovery opened my eyes to other pieces of 360 journalism where the reporter allows or encourages, the subject to tell their story direct to the 360 camera.

360 Case Study 1: Disfellowshipped

One of the most interesting of these is Disfellowshipped by Trey Bundy for the Californian publisher Reveal, in collaboration with the German Web VR start-up VRagments.

Bundy builds the interview around audio interviews with his main subject, a survivor of abuse in a group of Jehovah’s Witnesses in the United States. This audio is the backing to everyday 360 images of the subjects hometown, and a mix of newspaper cuttings and case records from over the years. And the really powerful moment comes when audio and 360 video become one, and the subject tells her story, straight to 360 camera, in her living room. And the special power of the scene is the incidental information we gather by witnessing the testimony told by the speaker in the home where she has somehow rebuilt her life. The small details of domesticity and order really tell.

This is how Trey Bundy describes the shot, and his whole first experience with 360.

So, what’s the point of telling this story in VR? The answer is to give the viewer a more intimate understanding of a character and her experience. The technology allows us to put you in the reporter’s shoes, to feel what it’s like to sit with people as they look you in the eye and tell you their story, to visit their towns and the places that affected their lives. In some instances, it becomes a window into a person’s emotional memory.

“Look you in the eye” tells it all.

In flat video you might ask the subject to look past the camera, to address the interviewer or a point of focus past the camera. In 360, you ask the subject to look down the lens, and in doing so you find the power of the format.

360 Case Study 2: The euronews experience

Euronews, based in Lyon, has led a ground-breaking programme (organised by Thomas Seymat) of reporting with 360-degree video, retraining their video journalists to add the format to their skill set. And their experience of reporting has shown the power of 360 in capturing incidental location information in a non-editorialising manner. (Declaration of interest: I have done a consulting report for euronews on their first set of news videos. It has been published on LinkedIn.)

After more than a year of the programme, the leader of euronews’s digital team Duncan Hooper posted a brief cri de coeur on LinkedIn that summarises the power of 360 as a journalistic tool, and that counters the received thinking (the one that I mentioned above) that demands that all 360 setting are spectacular, exotic or unworldly.

“There’s a school of thought,” Hooper writes, “that says 360° video only works if you are either showing fantastic images, or you are going places your audience couldn’t otherwise. This line of thinking says a 360° interview will almost never work unless you do it on top of a mountain or in the middle of a tomato-throwing festival. But sometimes the context of an interview does matter. Sometimes putting the audience in the same place as the interviewee can help them understand what that person is thinking.

“I would argue that interviewing someone inside their home, for instance, can be compelling in 360. Because as a witness to this interview I am interested in the casual details around someone’s house that tells me about them. From the books on the bookshelf to the Ikea lamp in the corner or the snow boots left lying about in the middle of the room. You cannot really show this in a flat video interview without endowing them with unwarranted significance.”

Hooper is right in everything he says.

And then there is Trey Bundy’s succinct comment on what made his first 360 journalism experience memorable: “to feel what it’s like to sit with people as they look you in the eye and tell you their story”.

That summarises why we at immersiv.ly have been working on one-to-one pieces with a variety of artists, creators and thinkers. To experiment with how people look you in the eye and tell you their story.

I will be talking about our approach in a master class organised by Raindance on 5 June. And we will be learning from Chris Timpson of Aurelia, one of the leading experts in the field, about why audio matters in 360.

Immersiv.ly will also be showing its content in the VR Studio at the Global Editors Network in Vienna from 21 to 23 June 2017.

Louis Jebb will be talking at the Raindance VR Masterclass: 1-to-1, the Power of Intimacy in VR + Live VR Shoot on Monday 5 June. Tickets available here

mm

About 

Louis Jebb is the CEO and founder of immersiv.ly, a London-based company that makes news content in virtual reality. He is also an ambassador for Journalism 360, an initiative of thought leaders, practitioners, and journalists dedicated to accelerating immersive storytelling in news. Immersivly Ltd received prototype backing in 2017 from Google Digital News Initiative for Svhere, a VR content platform that takes users to hyper-relevant related content using machine learning. Svhere launches in beta in August 2017.