Alternate States
Rory Fellowes
Issue: Volume 40 Issue 4: (Jul/Aug 2017)

Alternate States

VR pioneer Mark Bolas relayed this anecdote while we were discussing the development of virtual-reality and head-mounted display (HMD) technology. When he was a teenager, working in a photographer’s studio, he would closet himself in the darkroom, with all lights off, in the pitch black, and then would wiggle the fingers on his visually invisible hand, but force his brain to imagine a hand anyway.

This is the core of the spirit and vision of virtual-reality (VR) technology: the ambition to coerce the brain into believing it is seeing “real” worlds.

Virtual reality, augmented reality (AR), and mixed reality (MR) are the latest innovations in the tsunami of disruptive technologies that have surfaced during the last two or three decades as we have learned to live and work with computers.

Most people will have heard about these developments and possibly tried one of the HMDs available now while sampling the growing list of apps that are used with them. But we are in the early years of consumer VR gear, so expect to hear and see more head-mounted displays in the future. In fact, the current versions are already being replaced, and one day, perhaps soon, those, too, will be overtaken by technologies that researchers have only just begun to imagine.

These technologies will evolve from novelty to commonplace soon enough, just as the mobile phone did years ago. For now, they are a rarity, an indulgence for the enthusiasts. But, you can expect that within 20 years at most, the majority of people in advanced economies will use whatever HMDs have become by then – a device as light as a pair of wraparound Ray-Bans, maybe even just contact lenses, and capable of covering all one’s computer and communications needs. A one-stop telecom device people can carry in their pocket or wear over their eyes, or even plug directly into their brain, given time (research on this has already started).

INNOVATION

“The long nose of innovation.” That is how Bill Buxton, a principal scientist at Microsoft Research, describes the way all so-called breakthroughs result from many small, incremental developments over long periods of time before they come to the attention of the wider world. In the case of virtual-reality technology and software, that incremental process – from the earliest research and development of the dedicated hardware and software – has been going on for more than 25 years at least.

Alternate States
BUILDING CONVERSATION’S AR APP IS GEARED FOR THE CONSTRUCTION INDUSTRY, TO IMPROVE COMMUNICATION.

Currently, the quality of the computer graphics used with HMDs is not much better than the graphics imagery of late-1990s video games. But this is merely the stage we’re at, and it is rapidly improving all the time. How sophisticated those virtual environments may yet become, how “real” they may yet appear, and where they may take us as we become accustomed to them are questions already being discussed. Some tweaks of hardware, some major upgrades in Internet delivery, and there is no intrinsic, technological limit to what becomes possible.

Games, entertainment, new forms of narrative, virtual experiences, plus all the potential business uses are just now getting started. Over the next two or three decades, however, we are going to grow accustomed to virtual and mixed realities in our everyday lives.

Bolas has been given the nickname “the godfather of VR” by his peers. He is generally acknowledged as the most significant pioneer of research into virtual reality, analyzing the issues of representing a consistent 360-degree computer-generated environment that the brain would find believable. These days, Bolas describes himself as “a researcher exploring perception, agency, and intelligence.” He is director of the Mixed-Reality Laboratory at USC’s Institute for Creative Technology. He is also associate professor of interactive media at USC’s School of Cinematic Arts, and cofounder of Fakespace, which builds tools used by VR centers.

Bolas’s VR breakthrough came in 1988, when, as a student working on his Stanford design thesis, he received a place in NASA’s VIEW program, where the earliest efforts to design the related hardware and software were under development. According to Bolas, it was here, with the VIEW program, when the story of VR really began to take hold.

One could be forgiven for thinking that Oculus Rift was a major breakthrough in the development of VR. After all, Facebook paid upward of $2 billion for the company, but in fact much of the relevant hardware and software had been in development for some time at USC, in Bolas’s department. It was, and still is, open source, free for anyone who wants to make use of it.

“From my perspective, the real breakthroughs in reducing motion sickness and making VR accessible to a wide audience were about finding a way to do a wide field of view, with phone displays and with inertial tracking,” says Bolas. The wide field of view was the key to giving the viewer a convincing “immersive” experience.

Alternate States
LOW-COST HMDS (LEFT) HAVE SPARKED AR, VR, AND MR DEVELOPMENT. RIGHT: NDREAMS’ VR “GUNNER” GAME.

The seminal moment for the USC researchers, in Bolas’s estimation, was their discovery in 2012 of the UltraOptix 7x aspheric lenses, which provided a 90-degree field of view (FOV), the minimum needed for VR immersion. Alongside this was the Fakespace Labs’ Wide5 HMD, which actually provided a 120-degree FOV – more than has since been achieved by any of the consumer HMDs so far. Experimentation established that a 90-degree FOV was sufficient to create a convincing VR experience.

The ambition was to disrupt the industry by making VR affordable so everyone could enjoy it. For this reason, USC’s R&D work in the field is available for free, as open source.

The concept was catching the imagination of many, not least Palmer Luckey, who, in 2011, following publication of the lab’s work on the smartphone HMD, went to work at Bolas’s lab at USC. He was part of the team that came up with the FOV2GO Models A and D cardboard prototype, and Bolas suggested that Palmer utilize a Kickstarter campaign to further the concept. The rest, as they say, is history.

This is only a brief overview of a long and arduous process taking place over the last 30 years. Bolas and his many collaborators at his Mixed-Reality Lab (including USC Professor Perry Hoberman, an expert in the field of stereoscopy, and Thai Phan, MxR’s resident development engineer) were caught up in the excitement of research. But the rest of us might ask, what good is it? We can see the marvel of it, the fun, but in this world, the problem is always how to make it work for the consumer. How VR will become a consumer must-have (and it is surely destined to achieve that status) is for the entrepreneurs of the market to test and the consumers to decide.

Virtual Reality

The potential for VR in entertainment is already being exploited. Games, that major arena of innovation since the earliest days of CG, are bound to make great use of it –

after all, gamers obviously enjoy immersing themselves in virtual worlds. Dramas, comedies, thrillers… in fact, all forms of acting entertainment are a different story, as it may be more complicated to adapt an offering in this space to the uncertainties of a 360-degree environment.

One problem writers and designers are wrestling with is how to focus the audience on a particular scene of action. Another major consideration is how to accommodate all the possible scenarios as the audience wanders freely through the world of the movie – an issue already encountered by narrative game developers. At the very least, one must consider the vast amounts of data involved: The visuals and voice tracks that must be stored, sorted, and transmitted, not to mention created in the first place.

Is a viable market in sight?

nDreams

nDreams is a game development company founded in 2006 by CEO Patrick O’Luanaigh and now focusing on HMD apps. Its offices are in Farnborough, England, not far from London.

“We’ve done many brainstorms about where VR could go and what would be interesting, and you end up with pieces of paper that could cover a whole wall,”

O’Luanaigh says. “It goes into education, film, and TV, and then hotels and penthouse apartments. If you’re selling a house, you could actually wander around it! Then you have live broadcasts….” These are only the possibilities that spring to mind now, but as with all media, what is now generates what will be, and who can say where this technology may lead?

nDreams’ initial products are games, which are the most likely to generate revenue in the immediate future, company officials contend. According to O’Luanaigh, those who buy HMDs initially are “the sort of people who are into new technology. They’ve got the money.” They also tend to be gamers. This year the price of HMDs will fall dramatically, and that’s when a potentially much wider audience will begin to take on the idea of virtual experiences and virtual worlds. This wider public, O’Luanaigh says, will at least initially step into VR with experiences and environments that provide the opportunity to be somewhere other than here.

“We have this thing called ‘Perfect Beach,’ which is completely opposite of a game. You just chill and relax, you feel like you’re back on holiday for a few minutes, getting away from the craziness of the world,” O’Luanaigh says.

Jaunt

Another company already successfully exploiting the business potential of VR is Jaunt (Santa Monica, California), founded in 2013 by Arthur van Hoff and Jens Christensen, now company CTO, with George Kliavkoff as CEO/president. Jaunt has developed its own technology and product, in readiness for the coming revolution of VR as HMDs hit the wider markets of mainstream personal entertainment.

Alternate States
JAUNT IS DEVELOPING ITS OWN VR TECHNOLOGY FOR UNIQUE EXPERIENCES, SUCH AS THIS CONCERT.

In 2015 I spoke with Jaunt Studios’ then president Cliff Plumer. This was still a time when the industry was just beginning to ramp up. Plumer has since moved on to become CEO of the immersive entertainment company The Void.

Plumer came from the VFX industry, working closely with George Lucas on the first of the digitized series of Star Wars prequels, later assuming the role of CEO of Digital Domain. He saw 2015 as the year when VR was introduced to the content production community, allowing them to develop over the coming years the apps that will catch the public’s interest.

At the time, Plumer said Jaunt created its own versions of experiences for a variety of audiences, aiming to give the viewer the opportunity to observe an environment from a unique and otherwise inaccessible viewpoint. It provided VR tours for holiday planners, a vertiginous north face ascent alongside a fellow climber, and another that gives the viewer the opportunity to be on stage with Paul McCartney in concert.

Jaunt has moved on since those conversations, and at what I would call a considerable pace. Recently, Jaunt VR’s head of production, Scott Gemmell, discussed the launch of a number of HMD-delivered VR drama series, developed in conjunction with Hollywood writers and directors, and again shot with Jaunt’s proprietary camera technology.

At FMX 2015, creating dramas for VR was discussed by a panel headed by Professor Alex McDowell of USC’s World Building Institute, where this genre was seen as one of VR’s most difficult challenges, particularly how to direct audience attention along a coherent narrative. Gemmell is enthusiastic about the company’s ability to meet these brand-new challenges in creativity and technical production.

With only one series aired so far, Jaunt is still exploring and learning by experience, determining how to create material that will hold an audience, and Gemmell is confident the group is making solid progress. He made particular mention of heat mapping, software that allows them to track where the viewer is looking at any given moment, so as to prevent audiences from missing a crucial moment because they’re looking the wrong way.

But Gemmell suggests that’s simply old-fashioned movie thinking, and believes a new form of storytelling will emerge that defies such assumptions. As he says, there are the technical issues of shooting in a 360-degree studio – for instance, no personnel or equipment can be in the room. For Gemmell, though, that only points to solutions that involve 360-degree narratives so that the audience’s varying focus of attention can be met (which chimes with the World Building Institute’s research work).

Gemmell is confident of continued expansion in this market, with more and more product becoming available, and adds that his team is fired up at the possibilities of what they have only just begun.

At present, the episodes in the Jaunt series run 5 to 10 minutes each. The team was concerned about how long people would be comfortable wearing a tethered HMD, but “a lot of people binge-watched the whole five-episode series at a sitting,” says Gemmell, who hopes to produce 90-minute dramas in the near future.

As Plumer said back in 2015, “It is such a new medium. In the past, to launch a new platform, you could acquire a library of content. But with VR, there’s nothing out there that will work. We have to basically create it all ourselves.”

It seems they’ve already started.

Going Mobile

Oculus Rift, HTC Vive, Sony PlayStation VR, and the other head-mounted displays are tethered devices in that they attach to a desktop computer, requiring the user to be a cord-length away from the PC. Jaunt’s entire current product runs on these devices, but industry expectation seems to be that mobile is the future platform customers will demand.

Tethered devices are still working to build an audience in the tens or maybe hundreds of millions, but there are already more than two billion smartphone devices active in the world. And as the technology develops and as the screen resolution improves, it seems logical that mobile devices will be the preferred way to experience VR and all forms of media delivery.

Augmented Reality

Terrence Masson is the founder and CEO of Building Conversation, a company that has developed the Building Conversation AR app for a target audience of architects and other designers involved in the construction industry. The aim is to solve the problem plaguing all previous means of communication at their disposal, from blueprints to Photoshop composites, where, as Masson says, “you’re trying to explain a 3D concept in a 2D medium.”

Alternate States
PATRICK OSBORNE CRAFTED "PEARL," THE FIRST OSCAR-NOMINATED VR SHORT FILM.

The app aids the conversation between professionals and their clients, who often find visualizing the final building from traditional sources difficult or even impossible. The app – which runs on an iPad or smart device – basically allows a person to review a design on a turntable, akin to a modeling program. Used at the construction site, the app will geo-locate the model as a hologram in the landscape, fully textured and lit according to the time of day if desired. The structure also can be viewed as a simple polygonal model or at any resolution, with zoom/rotate and other features expected from these mobile devices.

Although Masson spoke mostly of the conversation that takes place between client and architect, clearly the application would work for designers, as well, enabling them to review their work in the real world.

The hologram mode is already familiar, where a flat 2D image on a wall or table surface will trigger the app to grow a 3D model from the AR image when viewed through the iPad, eliminating the need for foamboard models and offering alternatives to a client simply by toggling while viewing the hologram.

For now, Building Conversation is still a 2D experience on the user’s device screen. So, what about VR? On this, Masson is not yet convinced. It’s difficult to work with it on a construction site where the risk of stumbling is great – even assuming that the HMD is tetherless, which, as noted earlier, they seldom are at this time. He also notes that headsets are bulky and uncomfortable, and most people are not at ease wearing them.

At this time, those at Building Conversation are wary of the current wave of new devices, as Masson points to the short-lived life of Google Glass. They are waiting for a stable and widely adopted technology before taking their app beyond AR. They are aware of what may come, and “when it does, we will port to it,” Masson adds.

Mixed Reality

Mixed-reality HMDs place virtual objects – 3D holograms, if you will – within the world around the user. A person can place a TV screen (any size) on the wall or elsewhere and watch a show while walking around. Or, Skype a friend, surf the Web, or all of these at the same time, with the mechanisms placed by the user. It won’t be long before this capability becomes a fully operational telecom system, with a switch between mixed and full-immersive virtual reality.

It also includes the possibility of creating personal holograms. This means designers can work together on a single virtual object, walk around it, change it and its position in the real world, as solidly fixed as the ground beneath one’s feet.

Compared to VR, MR has the potential for a wider audience due to its more immediate practical use. Mobility and practicality are the hallmarks of consumer choice these days, the one that everyone from a 10-year-old to an octogenarian can relate to, given the appropriate apps. It took only 15 years for mobile phones to go from a rich man’s toy to a universal necessity. MR, once perfected, could follow the same pattern. At present the HoloLens and other similar MR devices have a limited field of view, something around 45 degrees, but in time, 180 degrees of vision will no doubt be perfected.

Canon has already released a mixed--reality device that retails for $125,000, which makes it more for business use than consumer consumption. Also, Microsoft’s HoloLens is launching a developer kit for $3,000. The consumer version of this device is still pending but cannot be far off. Magic Leap is not yet revealing a schedule or price for its MR device, but rumor is that it will astound us all over again with its unique approach to the problems of convincing MR imagery. Meanwhile, prices for the various MR and VR devices that have shipped are already falling as the market expands.

The new Oculus Rift and HTC Vive, developed in collaboration with Valve, have added haptic hand controls that allow for interaction. At present, the available apps are fairly simple and basic, but when this goes on to include fully interactive gaming or experiences of more detailed and engaging worlds, then it will have a wider appeal, offering ever more believable virtual experiences.

The Future

We humans seem able to adapt to all sorts of technological advances. After all, we have been doing so ever since we gave up living in caves and figured out how to build houses, which are, one could say, virtual caves.

Professor Ken Perlin is one of the early masters of CGI, first making his mark on the industry back in the 1980s as the original inventor of the procedural networks for creating and applying textures known as shaders. Nowadays he is a professor in the Department of Computer Science at New York University, the founding director of the Media Research Lab at NYU, and director of the NYU Games for Learning Institute.

Alternate States
NDREAMS IS OFFERING USERS AN ESCAPE TO VR ENVIRONMENTS . MICROSOFT'S HOLOLENS IS GEARED TO THE AR AND MR WORLDS.

Perlin and his team at NYU are currently working on a fully tetherless VR system, one that allows a person to live in a perceived virtual world while moving freely and safely through the physical world. This is done by providing “equivalence in the virtual world of the obstacles, the walls, furniture, the ‘curves,’ of the physical world.”

Imagine, for instance, walking down a crowded city street wearing a tetherless device. The people, traffic, and buildings would appear in the virtual world, but they might be aliens, with the buildings as cliff faces or the walls as some vast castle structure, while the traffic could be a river of floating vessels. It can be any type of virtual environment; the point is that the user is aware of the objects. “We’re not going to have people walk into physical walls and not see them,” says Perlin.

Perlin imagines a world where children will learn to live in this kind of adaptive virtual/physical reality. After all, children are already at ease with devices like mobile phones and iPads.

Adventurous as this sounds, it is only the beginning. There is a lot more to come, and that is only what we know of now. An implant that allows blind people to perceive the world as a virtual image transmitted directly to the visual cortex has already been developed and successfully tested. The concept of an implant that will create all the sensations of a believable virtual world – touch, smell, sight, and sound – delivered directly to the brain is already under discussion.

This may seem far-fetched, and it is probably some decades away, but it is no longer impossible. One day it will be part of our normal experience. As Perlin says, “We got used to the telephone.”

Until such time, there is an adjustment period. The brain is vulnerable to deceit, as our visual and audio senses rule our awareness of the world, as does touch and taste, all of which can trick our brain. So, it is not a big leap to get our brains to believe in virtual worlds. Plumer recalls an experience of wearing a VR headset that placed him on the roof of a skyscraper. “I walked to the edge of the roof and looked down at the street below. I thought about stepping off the edge. I knew I was in a room inside the building. I knew I was wearing the HMD, but I couldn’t take the step.”

And that is with the current resolution of 360-degree CGI, which is still a long way from the kind of convincing realities we expect from cinema HD VFX and high-end game consoles, let alone the resolutions of future devices. Everything you can imagine is possible. It’s only a matter of time.

Imagine that you are sitting in your dingy flat in a smoggy, overcrowded city somewhere, or you’re slogging your way to work through dense crowds. But with a VR device, you can experience a garden filled with flowers, trees, and grass gently blowing in the salted sea air. How long would you stay in that virtual paradise? How often would you escape into your VR device?

Rory Fellowes (roryfellowes@gmail.com) is a consultant to the VFX and animation industry following a career in film and TV animation using stop motion and CGI.