Digital Humans Becoming Reality

© 1982 The Walt Disney Company

It was 1982 when I first saw Flynn get digitized in the film “Tron” and transported to the Grid, an immersive world inside the computer. I was hooked immediately. Could it be possible to digitize humans in my lifetime? 

Fast forward 30+ years...

Last year, inspired by the world’s renewed interest in VR and AR, I decided to pursue a way to digitize real humans, objects, and environments so that they could be viewed from any angle in an immersive 3D space. Some very smart colleagues and I formed HEXXU CORP to accomplish this mission. When I saw the first moving tests of one of our digital humans in a virtual world, I was transported back to that influential scene from Tron.

YouTube link to Virtual Carl in Motion

YouTube link to Virtual Carl in Motion

Why Create Digital Humans?
As if turning Tron’s science fiction into science fact wasn’t a good enough reason, it turns out digital humans are on the verge of becoming big business - really big business. A new format for delivering live-action content for VR, AR, and mobile will have a disruptive and beneficial impact on many sizable industries including film, television, broadcast sports, Fantasy Sports, eSports, advertising, corporate telepresence, customer support, and social media.

Facebook Social VR demo at Oculus Connect

Facebook Social VR demo at Oculus Connect

As an example, you might have seen the amazing Facebook Social VR demo at the 2016 Oculus Connect Keynote with cartoonish avatars. Check it out if you haven’t seen it. Imagine if that same long distance personal connection could happen with a digital version of the real you instead of a cartoon representing you.

Since beginning this challenge, we’ve noticed several other companies validating the need for this while demonstrating the problems with different approaches. The consensus among them is to call this new experience Volumetric Live-Action. This is why engineers aren’t usually part of marketing. It doesn’t really roll off the tongue, but it does aptly describe what makes the various approaches similar. The ability to capture and play back live action events in a way that can be viewed from anywhere inside a given volume/space. 

What separates these companies and their approaches are their features and visual results. Some don’t even bother trying to look realistic. Some limit the viewer’s movement to just turning their head and leaning, but not really moving around. Some require minutes to process a single frame or hours to process a short clip. These are all important steps in the journey to volumetric live-action but currently they fall short of what should be possible with live-action VR/AR.

Suspension of Disbelief
HEXXU is the only company we know of that is actually capturing high-fidelity digital copies of real people and objects and giving the viewer unlimited 3D motion and personalized viewing angles while playing back motion in a recording or a live event. Our goal is to suspend disbelief and have the viewer think they are looking at a film or HD broadcast, with the added bonus of being able to move around inside it. 

Here’s another way to describe it. When your friend calls you on the telephone or messenger call, you don’t think about the fact that what you are hearing is a compressed digital signal originally sampled periodically over time while your friend talked. It sounds close enough to reality that you feel like you are hearing your friend’s voice and talking with them. HEXXU is doing the same thing with visual and motion quality of digital humans so that you believe you are looking at and talking with your friend even though you know they are 100 miles away. That is what differentiates HEXXU’s approach for digital humans.

I could go into a bunch of technobabble about how HEXXU eliminates the capture issues of occlusion, camera resolution, and depth sensors, but the bottom line is HEXXU’s patent pending technology simply looks better and more realistic, whether you are one inch or one hundred yards away. 

Up close and personal with HEXXU Virtual Carl

Up close and personal with HEXXU Virtual Carl

Are we there yet?
Phase 1 on our R&D schedule was about testing different scanning technologies and gaining the knowledge required to reproduce and automate the process of creating visually realistic looking digital humans. We are already suspending disbelief with a large percentage of the audience when using still images of our digital humans in testing. Can we fool 100% of the audience? No, not yet. What’s missing? In a word, motion. Communication, that personal connection, and believing a live person is there in front of you requires accurate and natural looking motion.

We spent much of this year researching and testing popular motion capture solutions on the market in industries such as film visual effects, game development, or simple live chats. While there are great solutions for each of those fields, we concluded there isn’t a single head-to-toe solution that works for where digital humans are headed. As a result, HEXXU is entering Phase 2 on our R&D schedule, developing a proprietary motion capture and playback technology.

An ENCOM Digitizing Laser in Every Home
We can’t all afford Tron’s multimillion dollar digitizing laser from ENCOM, not to mention it is a fictional device. But based on our first year of research and tests, HEXXU is moving in the right direction to develop a real, fast, and affordable technology that will make realistic digital humans a part of everyday life in every home. 

Tron Image: © 1982 The Walt Disney Company.

Be the Ball, Danny

In my previous blog post, God Mode for Live Action VR, I described our invention for digitizing real people performing in real environments from every angle. We do so with a high degree of visual fidelity while enabling every viewer to control a personalized view of the performance. This is so much more than 360° video where the viewer can only turn their head. With our Holographic Entertainment eXperience (HEX VR) the viewer can actually move around and explore a 3D space from any POV while watching live action.

Simulated HEX VR Rendering

Simulated HEX VR Rendering

I ended that blog post with a small bombshell that, in addition to film and TV production, the HEX VR technology can also be used in sports stadiums. It didn’t take long for us to put 2+2 together.

With HEX VR you can view a live sporting event while walking out onto the field virtually. Let’s use an NFL® football game as an example. I don’t mean you can just stand on the sideline, or sit in any seat of the front row of the stadium. Instead, you can stand on the field right behind the quarterback while he is snapping the ball, LIVE. Or you can see out of the eyes of any player on the field or sideline. Watch from the moving POV of the running back as he’s juking and spinning to find the gap in the line to shoot through. Be the safety closing in to deliver that hit and end a play. It brings an incredible feeling of kinetic energy watching a play like this. It’s like being in the game.

Wait, it’s Sunday and I don’t want to control my TV, phone, tablet or head mounted display to enjoy the game. No problem. With HEX VR you default to never before possible camera angles that can automatically keep you in the middle of the action. “Be the ball, Danny.” With HEX VR you can. When the QB throws a pass, you can watch the ball spiral in front of your face as you travel with it, arcing over the fingertips of the defensive secondary and into the hands of the receiver. My mind was blown when I realized we could do that.

Simulated HEX VR Rendering

Simulated HEX VR Rendering

What just happened? I want to see that play again! This is an area where HEX VR really shines bright. Use your mobile device or interactive remote to control the replay. You control moving time forward or backward, and at fast or incredibly slow speed. Another benefit to the way HEX VR works is that playback is not limited to the frames that were captured by a camera. Unlike today’s sports broadcasts, you can roll time forward or backward so slowly that the ball is moving only millimeters at a time.

Let’s not forget God Mode for Live Action means you are the master of both time AND space. With that incredible time replay capability, you are no longer limited by the camera angles of today’s sports broadcasts. I’ll repeat that. You control your own camera angle. Want a better view of what just happened on the other side of that player? Just move around, above, or behind him controlling the viewpoint of the replay. You can even look down and zoom in to see individual blades of grass on the turf.

What do impossible camera angles and player POV’s mean for experiencing live sports like football, basketball, baseball, soccer, tennis, and racing? That is the question we are working on every day while developing this invention. What would you do with these super powers? Feel free to comment on our Facebook page.

There are so many areas of entertainment, both live and scripted, that this invention can impact. Did I mention HEX VR sips bandwidth and is both mobile and Internet friendly? The live stream of this revolutionary viewing experience works at a fraction of the bandwidth required for today’s HD video streams. On that note... until the next blog post.

God Mode for Live Action VR

A funny thing happened at HEXXU on the way to creating groundbreaking augmented reality (AR) and virtual reality (VR) content...

Last year, after we released our Raptor Hunter simulated AR experience and optioned an incredible screenplay, we started attracting some stellar talent to our growing team. Every time we would get together to brainstorm we’d find an area of the pipeline that needed improvement, or in some cases invention, in order for this very new form of entertainment to become a reality.

The biggest issue for AR/VR is live action, as in being able to move around real humans in a real place virtually. As Michael Abrash, Chief Scientist at Oculus, put it in his recent blog post, “Perhaps the most important problem yet to be solved is figuring out how to represent real people convincingly in VR, in all their uniqueness. Other people are what we are most highly tuned to, because they are what we care about most – and for that same reason, representing them believably is one of the greatest challenges.”

To date 360° videos have been a popular way to experience live action in VR.  You can turn your head and look in any direction while watching a recorded video. That is very cool and an important step toward immersion and the feeling of virtual presence. Unfortunately, your feet are pinned to wherever the 360° camera rig was located when the video was recorded. You can look but you can’t move around or explore the space you are viewing. It quickly starts to feel restrictive and delivers only part of the AR/VR promise for immersion.

We needed a solution which could capture real people as they were moving/playing/acting in a way that every viewer could move around them with their own personalized view. A solution with the convincing visual fidelity that today’s sophisticated HD and 4K viewers expect. To borrow a term from the videogame industry we needed God Mode but for live action content. We looked around for an existing solution. We found that Microsoft was publishing some really cool 3D moving people with their Free Viewpoint Video project. We also noticed a company called 8i early in the stages of something similar. We identified another company called Uncorporeal that had a different approach but created similar visual results. Each company had developed an impressive and unique approach to capture one or two people at a time. But, we needed something bigger – much bigger. After all we had a VR film to make.

At HEXXU we wanted to capture an entire soundstage with all of the actors, props, and set pieces at the same time.  We wanted to capture a film or TV scene as it was performed, as close as possible to the way it is traditionally produced, but with no limitation on camera position. The viewer should be able to move around the set and determine their own personalized camera view after the scene is shot. The technology to do this did not exist. So we invented it.

Wait, what? We just invented a new way to shoot movies and TV that wasn’t possible before? Just like that?

Remember that talented team I mentioned? Our team embraces creativity with technology development as much as we do with art and production. We’re innovators with a long track record of creating new technology. We see obstacles as opportunities. We have received dozens of patents and technical achievement awards over the years. So we weren’t discouraged or deterred when we were told by industry leaders that live action for AR/VR on a large scale wasn’t possible. Instead, we jumped for joy and got to work.

We thought through and discarded several approaches using various technologies in new ways. Then it happened... We came up with a creative solution for some of the biggest problems blocking true live action capture. We invented God Mode for Live Action VR. We jumped for joy again, and then quickly filed patent applications to cover the new inventions.

So here we are with a completely new production technology that enables everyone to control their own personalized view of a movie or TV show.  We call it the Holographic Entertainment eXperience - HEX VR.

Every day we are having a blast thinking about what this means for the future of entertainment. It didn’t take long to realize some of the additional benefits of our invention. One of my favorites is that the performance of a real human can be slightly modified in a convincing and natural way, after the performance is shot, to dramatically improve connecting with the viewer. Instead of just virtually walking around a set and observing a film or TV scene, using HEX VR you can become part of the story. The lead character can actually turn toward you, even lock eye contact, and deliver a key line TO YOU, no matter where you are located in the scene. You can be in the story, a part of it, not just watching it. That changes everything for the future of interactive storytelling.

Just as soon as we came to grips with what that means, I mentioned to the team that this technology can work on a scale even larger than a soundstage. It can work with sports stadiums. I’ll just let that sink in and we’ll save that topic for the next blog post.

What is Mixed Reality?

You are seeing the terms Virtual Reality (VR) and Augmented Reality (AR) all over the place lately.  With Facebook’s acquisition of Oculus Rift, Google’s backing of Magic Leap, and Microsoft’s announcement of HoloLens, the world of VR and AR is quickly becoming a real part of our everyday lives.

In both VR and AR, the user wears a device on their face, or holds up a mobile device, like a phone or tablet.  The biggest difference between the two is that VR replaces the user's entire real view with a virtual view, while AR augments the user's real view.  In VR, this means the user can no longer see or interact with the real world around them.  The user’s view is replaced with a real-time rendered world, game, or virtual movie.

With AR, the user either wears a smaller visor-like device, or holds up a mobile device, that does not obscure their vision.  The user can still see and interact with the real world around them.  The device projects virtual objects onto the user’s field of view.  Some devices like smart glasses overlay information and images, such as maps or contact information, locked onto a specific point in the user’s view.  For example if the map was in the upper right of the user’s field of view and the user turns their head to the left the information remains in the upper right of the user’s view, as if the information is attached to the inside of the visor.

Recently, technology products were announced that enable a specific category of AR called Mixed Reality (MR).  You’re thinking, “Why do we need yet another two letter reality acronym?!”  Mixed Reality is special and you’ll see below why it really needs to be differentiated from other forms of AR.  Unlike informational AR such as smart glasses, MR devices have the processing power to track the user’s head and eye movements, render realistic looking 3D objects, and place them in the user’s real-world view.  This tracking and rendering happens in real-time, so there is no lag between when the user moves their head and when the objects are re-rendered and repositioned in space.  This means if a velociraptor is standing on the user’s coffee table and the user turns their head to the left, the raptor stays with the coffee table instead of staying in the same spot in the user’s view.  This enables the powerful illusion that the alien is actually located in the user’s real-world rather than just projected on a visor. 

Mixed Reality is hugely different than the rest of AR and it opens an entire world (infinite worlds actually) of possibilities for enhancing how users play games, interact with computers and data, and communicate.  With HEXXU it also changes the way users experience entertainment like movies, television, sports events, concerts, and social media.

That last part is where HEXXU comes in.  We refer to HEXXU as Mixed Reality Studios.  It is a new type of hybrid studio that merges the cinematic quality of feature films and visual effects (i.e. Movie Magic) with the cutting edge technology of high-end game development.  Both realms (movie technology and game technology) are required to create the believable and compelling experiences necessary to help drive mass adoption of Mixed Reality devices.  HEXXU challenges technological boundaries to create original content and experiences for Mixed Reality hardware.  At our core we are a team of experienced and passionate technologists and creative artists... think early days of Pixar but for a new Mixed Reality world.

The term Mixed Reality has been around for a couple of decades, but we are just now reaching the point where technology can deliver on the promises of this experience.  Combined with compelling content like our Raptor Hunter experience this is rapidly becoming an exciting time for the world of entertainment and fans of movies and television everywhere.