God Mode for Live Action VR

A funny thing happened at HEXXU on the way to creating groundbreaking augmented reality (AR) and virtual reality (VR) content...

Last year, after we released our Raptor Hunter simulated AR experience and optioned an incredible screenplay, we started attracting some stellar talent to our growing team. Every time we would get together to brainstorm we’d find an area of the pipeline that needed improvement, or in some cases invention, in order for this very new form of entertainment to become a reality.

The biggest issue for AR/VR is live action, as in being able to move around real humans in a real place virtually. As Michael Abrash, Chief Scientist at Oculus, put it in his recent blog post, “Perhaps the most important problem yet to be solved is figuring out how to represent real people convincingly in VR, in all their uniqueness. Other people are what we are most highly tuned to, because they are what we care about most – and for that same reason, representing them believably is one of the greatest challenges.”

To date 360° videos have been a popular way to experience live action in VR.  You can turn your head and look in any direction while watching a recorded video. That is very cool and an important step toward immersion and the feeling of virtual presence. Unfortunately, your feet are pinned to wherever the 360° camera rig was located when the video was recorded. You can look but you can’t move around or explore the space you are viewing. It quickly starts to feel restrictive and delivers only part of the AR/VR promise for immersion.

We needed a solution which could capture real people as they were moving/playing/acting in a way that every viewer could move around them with their own personalized view. A solution with the convincing visual fidelity that today’s sophisticated HD and 4K viewers expect. To borrow a term from the videogame industry we needed God Mode but for live action content. We looked around for an existing solution. We found that Microsoft was publishing some really cool 3D moving people with their Free Viewpoint Video project. We also noticed a company called 8i early in the stages of something similar. We identified another company called Uncorporeal that had a different approach but created similar visual results. Each company had developed an impressive and unique approach to capture one or two people at a time. But, we needed something bigger – much bigger. After all we had a VR film to make.

At HEXXU we wanted to capture an entire soundstage with all of the actors, props, and set pieces at the same time.  We wanted to capture a film or TV scene as it was performed, as close as possible to the way it is traditionally produced, but with no limitation on camera position. The viewer should be able to move around the set and determine their own personalized camera view after the scene is shot. The technology to do this did not exist. So we invented it.

Wait, what? We just invented a new way to shoot movies and TV that wasn’t possible before? Just like that?

Remember that talented team I mentioned? Our team embraces creativity with technology development as much as we do with art and production. We’re innovators with a long track record of creating new technology. We see obstacles as opportunities. We have received dozens of patents and technical achievement awards over the years. So we weren’t discouraged or deterred when we were told by industry leaders that live action for AR/VR on a large scale wasn’t possible. Instead, we jumped for joy and got to work.

We thought through and discarded several approaches using various technologies in new ways. Then it happened... We came up with a creative solution for some of the biggest problems blocking true live action capture. We invented God Mode for Live Action VR. We jumped for joy again, and then quickly filed patent applications to cover the new inventions.

So here we are with a completely new production technology that enables everyone to control their own personalized view of a movie or TV show.  We call it the Holographic Entertainment eXperience - HEX VR.

Every day we are having a blast thinking about what this means for the future of entertainment. It didn’t take long to realize some of the additional benefits of our invention. One of my favorites is that the performance of a real human can be slightly modified in a convincing and natural way, after the performance is shot, to dramatically improve connecting with the viewer. Instead of just virtually walking around a set and observing a film or TV scene, using HEX VR you can become part of the story. The lead character can actually turn toward you, even lock eye contact, and deliver a key line TO YOU, no matter where you are located in the scene. You can be in the story, a part of it, not just watching it. That changes everything for the future of interactive storytelling.

Just as soon as we came to grips with what that means, I mentioned to the team that this technology can work on a scale even larger than a soundstage. It can work with sports stadiums. I’ll just let that sink in and we’ll save that topic for the next blog post.