Virtual Reality Viewing Comes to Netflix — and Soon to Hulu, Vimeo, Twitch, Facebook and Others

Most people think of virtual reality programming as being limited to video games or other immersive short clips. But, as various Internet content services have revealed, movies and even live events programming are also high on the list for VR treatment as well.

VR Content Providers

VR Content Providers

The next year may be the time when we learn whether or not VR programming catches on with large numbers of viewers, or whether it is just another passing fad.

This fall, VR is ready for market. Most of the news came out of the Oculus Connect 2 conference in Los Angeles, where a new Gear VR headset — priced at $99 — was announced. That low-cost headset, to be used with Android phones, and the mobile VR solution behind it was co-produced by electronics giant, Samsung.

Gear VR Headset

Gear VR Headset

The first viewing site out of the gate, Netflix, introduced a new free app that allows the viewing of its films in VR. CNN announced it will produce a live VR stream of the Democratic presidential debate on October 13. It will team up with NextVR, a live VR production company, allowing users of Samsung’s GearVR to experience politics like never before.

Coming online very soon will be Hulu, Vimeo and Twitch. Hulu, which will debut its VR this fall, said it will stream both its existing 2D content and short-form content made just for VR. The services are expected to be a bit like Oculus Cinema, where users can watch films in a virtual theatre.

Twitch viewers will be able to chat and comment on gaming streams. A slide at the Connect 2 event also hinted that Facebook, Fox, Lionsgate and TiVo will be joining the VR lineup soon, though no specific plans were announced.

Anthony Park, vp of engineering at Netflix, said his service had been working with Oculus, maker of the Rift VR viewer — due in the first quarter of next year — and the Gear VR, to develop the Netflix app for Samsung Gear VR.

The app includes Netflix Living Room, which allows members to get the Netflix experience from the comfort of a virtual couch. It works wherever they bring their Gear VR headset.

John Carmack, Oculus CTO

John Carmack, Oculus CTO

“We’ve been working closely with John Carmack, CTO of Oculus and programmer extraordinaire, to bring our TV user interface to the Gear VR headset,” Park wrote in the Netflix blog.

Carmack wrote a technical description of the process of turning Netflix’s video into VR viewable content in a long entry in the blog.

“Despite all the talk of hardcore gamers and abstract metaverses, a lot of people want to watch movies and shows in virtual reality,” Carmack wrote. “In fact, during the development of Gear VR, Samsung internally referred to it as the HMT, for ‘Head Mounted Theater.’ Current VR headsets can’t match a high end real world home theater, but in many conditions the ‘best seat in the house’ may be in the Gear VR that you pull out of your backpack.”

Carmack explained how he took the Netflix TV codebase and presented it on a virtual TV screen in VR.

Netflix Program Flow Chart

Netflix Program Flow Chart

“I wrote a ‘VR 2D Shell’ application that functioned like a very simplified version of our Oculus Cinema application; the big screen is rendered with our peak-quality TimeWarp layer support, and the environment gets a neat dynamic lighting effect based on the screen contents. Anything we could get into a texture could be put on the screen.

The core Netflix application uses two Android Surfaces – one for the user interface layer, and one for the decoded video layer,” he wrote. “To present these in VR, I needed to be able to reference them as OpenGL textures, so the process was: create an OpenGL texture ID, use that to initialize a SurfaceTexture object, then use that to initialize a Surface object that could be passed to Netflix.

“For the UI surface, this worked great — when the Netflix code does a swapbuffers, the VR code can have the SurfaceTexture do an update, which will latch the latest image into an EGL external image, which can then be texture mapped onto geometry by the GPU.”

The video surface was a little more problematic, Carmack wrote. To provide smooth playback, the video frames are queued a half second ahead, tagged with a “release time” that the Android window compositor will use to pick the best frame each update.

The SurfaceTexture interface that I could access as a normal user program only had an “update” method that always returned the very latest frame submitted. This meant that the video came out a half second ahead of the audio, and stuttered a lot.

“To fix this, I had to make a small change in the Netflix video decoding system so it would call out to my VR code right after it submitted each frame, letting me know that it had submitted something with a particular release time. I could then immediately update the surface texture and copy it out to my own frame queue, storing the release time with it. This is an unfortunate waste of memory, since I am duplicating over a dozen video frames that are also being buffered on the surface, but it gives me the timing control I need.”

Initially, he said, input was handled with a Bluetooth joypad emulating the LRUD / OK buttons of a remote control, but it was important to be able to control it using just the touchpad on the side of Gear VR. Our preferred VR interface, he said, is “gaze and tap,” where a cursor floats in front of you in VR, and tapping is like clicking a mouse.

For most things, this is better than gamepad control, but not as good as a real mouse, especially if you have to move your head significant amounts. Netflix has support for cursors, but there is the assumption that you can turn it on and off, which we don’t really have.

We wound up with some heuristics driving the behavior. I auto-hide the cursor when the movie starts playing, inhibit cursor updates briefly after swipes, and send actions on touch up instead of touch down so you can perform swipes without also triggering touches. It isn’t perfect, but it works pretty well.

For viewing, he said forget 4K or even full high-definition resolution. Because of content protection in the HD images, he said the resolution is now limited to 720x480.

“That is slightly lower than I would have chosen if the need for a secure execution environment weren’t an issue, but not too much,” he wrote. “Even at that resolution, the extreme corners are doing a little bit of pixel blending.

“In an ideal world, the bitrate/resolution tradeoff would be made slightly differently for VR,” he wrote. “On a retina class display, many compression artifacts aren’t really visible, but the highly magnified pixels in VR put them much more in your face. There is a hard limit to how much resolution is useful, but every visible compression artifact is correctable with more bitrate.”

NextVR production camera

NextVR production camera

Finally, since Samsung’s VR system uses Android phones for operation, Carmack noted that watching a two-hour movie in VR uses twice as much power as the standard Netflix app. Fixing that required quite a bit of optimization.

From Carmack’s technical description, it is clear VR for video is in its early days of development for video production. It will be up to consumer usage to drive the VR trend further or relinquish it to the trash heap of technological history. Only time will tell.

You might also like...

AJA Video Provides Flexible Bridge From SDI To NDI And Back Again

As broadcast facilities and other organizations that use media to educate and inform continue to carefully make the move to video over IP, they currently face two main options, with a range of others in the wings. ​They may opt f…

HDR: Part 39 - Creative Technology - The Good Old Days

Progress inevitably comes with compromise. We can’t complain about the technology that’s brought us from hundred-kilo tube cameras to the 4K cellphones of today, but anyone who remembers the fuzzy old days of the 1990s might also remember…

HDR: Part 38 - Creative Technology - Where’s My Jetpack?

At one time, it was traditional to complain about the things that science had promised us but failed to deliver. Now we actually have a jet pack that’s practical, or at least that lasts more than thirty seconds, we’…

The Technology Really Doesn’t Matter, It’s The Partnership That Lasts!

The Color (Or Resolution) Of Money - We live in a wonderful time for the creative media industry. Huge change processes are currently on-going, driven by technological and social factors - all bringing opportunities and threats in equal measure. Given…

HDR: Part 37 - Creative Technology - Producers Guide To Large Format

Cinematography is not generally a field given to the idea that less is more. Probably the most direct and current expression of that is large format, a term that isn’t even particularly well-defined, let alone a technology that usually p…