Virtual Reality Viewing Comes to Netflix — and Soon to Hulu, Vimeo, Twitch, Facebook and Others

Most people think of virtual reality programming as being limited to video games or other immersive short clips. But, as various Internet content services have revealed, movies and even live events programming are also high on the list for VR treatment as well.

VR Content Providers

VR Content Providers

The next year may be the time when we learn whether or not VR programming catches on with large numbers of viewers, or whether it is just another passing fad.

This fall, VR is ready for market. Most of the news came out of the Oculus Connect 2 conference in Los Angeles, where a new Gear VR headset — priced at $99 — was announced. That low-cost headset, to be used with Android phones, and the mobile VR solution behind it was co-produced by electronics giant, Samsung.

Gear VR Headset

Gear VR Headset

The first viewing site out of the gate, Netflix, introduced a new free app that allows the viewing of its films in VR. CNN announced it will produce a live VR stream of the Democratic presidential debate on October 13. It will team up with NextVR, a live VR production company, allowing users of Samsung’s GearVR to experience politics like never before.

Coming online very soon will be Hulu, Vimeo and Twitch. Hulu, which will debut its VR this fall, said it will stream both its existing 2D content and short-form content made just for VR. The services are expected to be a bit like Oculus Cinema, where users can watch films in a virtual theatre.

Twitch viewers will be able to chat and comment on gaming streams. A slide at the Connect 2 event also hinted that Facebook, Fox, Lionsgate and TiVo will be joining the VR lineup soon, though no specific plans were announced.

Anthony Park, vp of engineering at Netflix, said his service had been working with Oculus, maker of the Rift VR viewer — due in the first quarter of next year — and the Gear VR, to develop the Netflix app for Samsung Gear VR.

The app includes Netflix Living Room, which allows members to get the Netflix experience from the comfort of a virtual couch. It works wherever they bring their Gear VR headset.

John Carmack, Oculus CTO

John Carmack, Oculus CTO

“We’ve been working closely with John Carmack, CTO of Oculus and programmer extraordinaire, to bring our TV user interface to the Gear VR headset,” Park wrote in the Netflix blog.

Carmack wrote a technical description of the process of turning Netflix’s video into VR viewable content in a long entry in the blog.

“Despite all the talk of hardcore gamers and abstract metaverses, a lot of people want to watch movies and shows in virtual reality,” Carmack wrote. “In fact, during the development of Gear VR, Samsung internally referred to it as the HMT, for ‘Head Mounted Theater.’ Current VR headsets can’t match a high end real world home theater, but in many conditions the ‘best seat in the house’ may be in the Gear VR that you pull out of your backpack.”

Carmack explained how he took the Netflix TV codebase and presented it on a virtual TV screen in VR.

Netflix Program Flow Chart

Netflix Program Flow Chart

“I wrote a ‘VR 2D Shell’ application that functioned like a very simplified version of our Oculus Cinema application; the big screen is rendered with our peak-quality TimeWarp layer support, and the environment gets a neat dynamic lighting effect based on the screen contents. Anything we could get into a texture could be put on the screen.

The core Netflix application uses two Android Surfaces – one for the user interface layer, and one for the decoded video layer,” he wrote. “To present these in VR, I needed to be able to reference them as OpenGL textures, so the process was: create an OpenGL texture ID, use that to initialize a SurfaceTexture object, then use that to initialize a Surface object that could be passed to Netflix.

“For the UI surface, this worked great — when the Netflix code does a swapbuffers, the VR code can have the SurfaceTexture do an update, which will latch the latest image into an EGL external image, which can then be texture mapped onto geometry by the GPU.”

The video surface was a little more problematic, Carmack wrote. To provide smooth playback, the video frames are queued a half second ahead, tagged with a “release time” that the Android window compositor will use to pick the best frame each update.

The SurfaceTexture interface that I could access as a normal user program only had an “update” method that always returned the very latest frame submitted. This meant that the video came out a half second ahead of the audio, and stuttered a lot.

“To fix this, I had to make a small change in the Netflix video decoding system so it would call out to my VR code right after it submitted each frame, letting me know that it had submitted something with a particular release time. I could then immediately update the surface texture and copy it out to my own frame queue, storing the release time with it. This is an unfortunate waste of memory, since I am duplicating over a dozen video frames that are also being buffered on the surface, but it gives me the timing control I need.”

Initially, he said, input was handled with a Bluetooth joypad emulating the LRUD / OK buttons of a remote control, but it was important to be able to control it using just the touchpad on the side of Gear VR. Our preferred VR interface, he said, is “gaze and tap,” where a cursor floats in front of you in VR, and tapping is like clicking a mouse.

For most things, this is better than gamepad control, but not as good as a real mouse, especially if you have to move your head significant amounts. Netflix has support for cursors, but there is the assumption that you can turn it on and off, which we don’t really have.

We wound up with some heuristics driving the behavior. I auto-hide the cursor when the movie starts playing, inhibit cursor updates briefly after swipes, and send actions on touch up instead of touch down so you can perform swipes without also triggering touches. It isn’t perfect, but it works pretty well.

For viewing, he said forget 4K or even full high-definition resolution. Because of content protection in the HD images, he said the resolution is now limited to 720x480.

“That is slightly lower than I would have chosen if the need for a secure execution environment weren’t an issue, but not too much,” he wrote. “Even at that resolution, the extreme corners are doing a little bit of pixel blending.

“In an ideal world, the bitrate/resolution tradeoff would be made slightly differently for VR,” he wrote. “On a retina class display, many compression artifacts aren’t really visible, but the highly magnified pixels in VR put them much more in your face. There is a hard limit to how much resolution is useful, but every visible compression artifact is correctable with more bitrate.”

NextVR production camera

NextVR production camera

Finally, since Samsung’s VR system uses Android phones for operation, Carmack noted that watching a two-hour movie in VR uses twice as much power as the standard Netflix app. Fixing that required quite a bit of optimization.

From Carmack’s technical description, it is clear VR for video is in its early days of development for video production. It will be up to consumer usage to drive the VR trend further or relinquish it to the trash heap of technological history. Only time will tell.

You might also like...

Tech + Media Predictions 2021 - Part 1

The industry experienced futureshock head-on in 2020. The impact will take a long time to unwind but it’s already clear that some changes will be profound and not all of them bad. These changes include remote workflow permanency, virtual production s…

A Centralized Streaming Gateway For Live SDI Production And IP Distribution

In a time of social distancing, video professionals have turned to technology that allows them to work remotely yet collaboratively over a secure Internet connection. This remote production strategy has helped production and postproduction companies as well as video streaming,…

Sports Re-Starts Without Fans And The Virtual Crowd Goes Wild

In mid-May of this year, as countries such as Germany, England, and Spain considered easing COVID-19 restrictions to allow professional sports to resume, various professional sports leagues began discussions with broadcasters and production companies on the best way to televise…

Presented In Cinemascope

Electronic camera manufacturers have spent – by some measures – something like the last twenty years trying to make digital cameras that shoot pictures that look like real movies. Now, they’re making cameras with larger and larger sensors, the better to simul…

Moving Cameras

The world’s oldest surviving motion picture, often called Roundhay Garden Scene, does not include any camera movement. It’d be tricky to imagine anything approaching a move, since the scene, which was shot in 1888 by Louis Aime Augustin Le Pri…