New Realities: VR, AR, MR, and the Future of Design

BY HIMANSHU BHARADWAJ – DESIGNER @ TOPTAL  (Original article link here)

The digital and technological landscape is constantly changing, and in many ways accelerating. Designers tasked to come up with innovative ideas have to keep track of what is trending and where the creative opportunities are. Recognizing the changes that are occurring in the industry will help designers design smarter and make more informed creative decisions.

Bell curve graphic of the technology adoption life cycle

In the technology adoption life cycle, “Innovators” are the smallest group, yet the most successful. They are followed by “Early Adopters,” then “Early Majority,” and later, “The Majority,” or everyone else. As innovators, great designers usually fall into the “Early Adopters” category—big risk takers in uncharted territory. A great designer is also a visionary with the ability to think strategically and analyze trends in order to design better products or services.

Let’s look at some creative opportunities and emerging technologies and why, in order to design the future, designers need to think beyond the screen into new realities.

Artificial intelligence (AI) is giving a computer or computer-controlled device the ability to perform intelligent tasks such as reasoning or learning from experience. Cognitive computing was the term used when it all started a decade ago—the idea that, facing cognitive overload, we are going to need assistance making decisions. Citing obvious advantages of current artificial intelligence in use, we are already relying on AI digital assistants like Siri and Alexa (among others) to help us make simple decisions.

Autonomous agents make up the living network in enterprise – Information Age

Product designers will have an increasing volume of data information to inform their design decisions. Artificial intelligence experts leverage AI technology to create dynamic user interfaces, personalizing each user experience by analyzing every interaction within an app or a website. In the future, computers will get even better at making data-based decisions to improve automated tasks and our human decision-making processes.

With IBM’s Watson leading the way for companies like General Motors, Condé Nast, GSK, Citigroup, and Under Armour, the future of AI is going to be all about making complex and important decisions efficiently.

As creative professionals, we can use such ideas across multiple industry verticals. For example, The North Face used Watson to help create an intuitive eCommerce interface that helps customers make optimal purchase decisions.

Virtual assistants like Siri, Alexa, Cortana, Watson, and others rely on artificial intelligence to help do everything from finding a restaurant to sending an email or scheduling a meeting. Many of these are operated by voice, a more natural interface for humans than a traditional visual UI of windows and buttons. These technologies are likely to reshape the current design of search engines.

Voice-controlled virtual assistants – image credit: Amazon Echo, Google Home, Apple HomePod

Speaking is a natural and straightforward method of interaction for humans. Thus, it’s no surprise that voice-controlled virtual assistants have received a favorable adoption rate in our homes. Users can simply talk to the system rather than “operate” it with hands or gestures. For example, with an iPhone, using only one quick request, Siri allows us to transfer money via PayPal to another person.

As AI is integrated into apps, it will help intelligent virtual assistants (IVAs) become more and more intelligent. Nevertheless, regardless of how smart IVAs get, users with poor eyesight or other physical disabilities—who use screen readers and other assistive technologies—could still face significant challenges. It will be up to UX designers working with user experience researchers to bridge such critical accessibility gaps.

The transit systems of the future will have traffic flow through our streets smoothly and efficiently. Design innovations like modular, detachable buses, flying taxis, and networks of suspended magnetic pods will make the dream of a dynamic, driverless world into a reality.

Self-driving store – via dezeen

What was science fiction news some years ago is now becoming a reality. Wheelys’ Moby-Mart is a self-driving store with a holographic shop assistant that is likely to change the world of eCommerce design.

The next generation of eCommerce designers will need multi-specialization in both physical and digital spaces. Future designers will need to design an experience where many of the familiar human elements are removed or remote.

Amazon Go, a cashier-less store, is a great example of what designing the future looks and feels like. They designed a shopping experience where anyone can enter the store, browse, shop, and leave with their products in hand, friction-free. No lines at the checkout.

Driverless car concept by Mercedes-Benz

The intelligence built within each autonomous vehicle allows it to make all the right decisions for us—everything from the most optimal routing to human safety concerns. The creation of such complex design systems is creating a product template that will start a new wave of products designed outside the transportation vertical.

It will become necessary for automotive interior designers to create designs making interior space comfortable and adaptive. The interfaces will need to be intuitive and reassuring in order to encourage trust from the passenger while focusing on human interactions between people sharing the ride.

Augmented reality (AR) acts as a virtual layer on top of the world in front of you, and is in a new age of discovery and innovation. While early AR applications depended on a smartphone or tablet, they will soon expand into wearable devices like smart glasses requiring a different kind of UX and design process—thinking outside the confines of a smaller screen.

Apple Animoji demo presentation at WWCD17

While the debate over AR vs. VR vs. MR is ongoing, augmented reality is a unique opportunity for designers to create apps using image, object, and color recognition. Just like AI leverages data to make decisions, AR leverages the recognition of key “anchor” objects and points within a given space.

A great example of a perfect combination between AI and AR is with Apple’s Animoji, animated emoji for the iPhone X. According to Apple, Animoji is “custom animated messages that use your voice and reflect your facial expressions.” The rendering and animation work in real time, thanks to iPhone X ace-scanning features.

Meta glass AR demo – www.metavision.com

In the next five years, AR will be better integrated with our everyday lives, merging our physical and digital worlds with new wearable products. A product like Meta glasses, which overlays augmented reality on top of the user’s reality, is likely to be popular as a medium for game designers.

Based on holographic technology, Meta glasses can identify users’ gestures to let them manipulate 3D projections of objects. With so many devices based on gestures, standardization and cultural gaps could become a main challenge for designers working in the field.

Meta AR Workspace demo

“The future of productivity is spatial,” said Meta when introducing their AR Workspace. In the future, wearable hardware will enable employees to work hands-free. The devices of the future could work on voice commands or gestures. The increasing success of AR experiences may predict the end of physical controllers such as a mouse cursor or a keyboard.

In augmented reality, natural gestures could be the key drivers of the experience. For example, mimicking the click of a button with our fingers would trigger an interaction with a virtual link or button. Google launched project Soli, which goes a step further—they created a sensing technology focused solely on touchless interactions. To stay user-friendly, such experiences call for a minimal interface, with little to no visual overload.

Some notable augmented reality development resources:

• Facebook Frame and AR Studio
• Apple ARkit

As part of its concept kitchen, IKEA is developing a smart table that suggests recipes based on the ingredients placed on it. This is a great example of AR technology working in the real world. The interactive table has a camera-equipped projector that shows recipes on its surface and recognizes ingredients, giving you an idea of what to make with what you have on hand.

IKEA AR Concept Kitchen – image credit: RumbleRum

Such immersive experiences are in direct competition with mobile apps that will soon seem outdated due to their more restricted capabilities. Mobile app designers will soon be creating augmented applications that integrate with physical spaces to deliver a new and exciting way of interacting with our physical environments.

Augmented reality can also be used for knowledge transfer, such as training in industrial environments. Instead of explaining, imagine showing a worker how to do a specific task via in-context video. This opens doors for designers in the e-learning area to create next-generation AR interfaces.

Boeing uses AR glasses powered by Skylight to guide technicians as they wire hundreds of planes a year. The wearable display helps technicians identify and accurately connect hundreds of wires using only their sight and voice to control the app. As a result, Boeing cut production time by 25% and lowered error rates to nearly zero.

AR glasses powered by Skylight

A simple voice control system decreases the interaction cost to perform complex tasks when operating AR applications. Minimizing interface elements reduces a user’s cognitive load, minimizes interruptions, and reduces distractions.

Augmented reality technology could change the way customers and workers are trained and introduced to new information. Imagine training modules like how to change oil in a car, or other “How to” training videos using AR to layer indications on top of a real-life situation.

From a usability perspective, designers need to consider not only what the system presents to the user visually, but also how the brain interprets complex information overlapping the actual reality.

Virtual reality (VR) completely immerses you in a virtual world, and mixed reality (MR), sometimes referred to as hybrid reality, is the merging of real and virtual worlds. VR can be used for virtual offices, products, events like the 2017 Teen Choice Awards, and more. With VR, designers and their clients will be able to preview designs scaled to real, life-size environments and depth, before even entering the production phase.

In early 2017, Forbes observed that “the hype around virtual reality is fading.” Noticeable limitations VR faces are a high price for the headsets, discomfort over extensive use, poor visual quality, and much more. As a result, there is a slow mass-market adoption of VR technology in comparison to AR that is “likely to fare better in the short term.”

The novelty of virtual reality content offers a unique opportunity to engage large audiences as demonstrated by a VR documentary from National Geographic: The Protectors, Walk in the Ranger’s Shoes. It received more attention compared to other nature themed documentaries.

Movie poster from National Geographic VR Documentary film: The Protectors

Beyond transporting viewers into new places, the next opportunity for designers is helping them experience life as another person. Offering a first-person view can create unique and effective campaigns like “A Walk Through Dementia,” centered on empathy.

A new set of design patterns will ultimately emerge for VR designers, but a flat design approach is unlikely to succeed. In VR, our brain uses spatiotemporal orientation and problem-solving capabilities; therefore, it’s essential to use textures, lighting, and finer details for a more believable and immersive experience.

VR gives users the ability to get up close and personal with virtual objects and also move back from them. Since the eye is very good at picking out depth information, rendering of objects by designers needs to be realistic and detailed. Sound and music are other important considerations for VR designers.

Microsoft HoloLens mixed reality blends 3D holographic content with the physical world, giving holograms real-world context and scale. MR technology mixes and blends a person’s physical surroundings with objects from the physical and digital world together. Using this MR technology, we can interact with both digital content and the world around us as well as interact with 3D holographic projections.

Microsoft HoloLens interactive holographic projection

MR and VR applications give new creative freedom to designers. We can create experiences where users can interact with products virtually and visit distant environments, historical events, concerts, and much more. Within the virtual or mixed realities that we create, we can bend the laws of physics with endless possibilities and experience the world like never before.

From a design and usability standpoint, a key principle to keep in mind is to put the users at ease and avoid discomforting sensations by keeping them grounded in the real world. To that end, we can leverage sound, motion, and a good sense of scale—the virtual environments and interactions we create should feel natural and ergonomic, not necessarily realistic.

When designing a visual interface that uses a headset and needs a close examination of an object like a VR organ, we must keep in mind that people are much more susceptible to motion sickness when wearing a VR headset than when looking at a computer screen.

VR creates an opportunity for companies to provide a more immersive experience of their brand for customers. It’s a new frontier for branded content and storytelling. Audi uses VR so they can experience a new model of their car before even physically touching it. By capturing the full interior of the vehicle with a special 3-D camera, they allow customers see every detail before market launch.

An effective use of virtual reality marketing allows brands to leverage the following design components:

1. A well thought-out creative strategy
2. Application of usability best practices
3. Strong narrative and storytelling
4. Ergonomic information presentation
5. Compelling visual design and simulations

The definition of usability will be greatly expanded for designers and cover all aspects, including the comfort level of the environment. Designers will need to develop an understanding of the level of potential discomfort in certain situations like heights, small spaces (claustrophobia), big spaces (agoraphobia), scale, speed, colliding objects, user fatigue, ergonomics, and more.

Toyota VR Experience demo

To communicate the reliability and safety of their vehicles in difficult situations, Toyota immerses customers in a narrative VR experience. They can experience the brand as well as unique customized content in different environments. This opens the door for companies to show customers a side of the brand they haven’t seen before.

As designers deeply involved in creating such experiences, we must question the usefulness of our creative ideas and solutions for brands and make sure we’re truly solving specific business problems. We should be asking ourselves if the VR experience delivers on the company’s basic value proposition.

To create more empathy and contextualize the lives of those in need for donors, some non-profit organizations are using VR to move visitors into life inside refugee camps. This creates increased engagement and more willingness to support their initiatives. An idea like this could create a shift in the VR design work done by large brands into a non-profit space.

VR refugees camp 3D tour – International Society for Presence Research

As another example, Mark Zuckerberg showed the world a VR tour of hurricane-struck Puerto Rico in order to raise awareness of the situation. The ideas and use cases are “virtually limitless”—pun intended.

Using a virtual reality headset and a remote controller, users can explore the environment by moving their heads and interact with objects by using their hands. High-end VR systems like HTC Vive or Oculus Rift use sensors to locate users in physical space and, as such, designers can take advantage of the user’s entire body to craft fuller, more enhanced sensory experiences in the real world.

Recent developments in AI, particularly deep learning, that foster real-time image and speech recognition have helped combine VR and AI for new applications. The future of product design will need to account for the combination of all three of these technologies. Notable examples are Eolian applications that use AI technologies to reduce human error rates through AR and VR simulations of dangerous tasks.

Virtualitics provides data visualization in VR and AR environments through machine learning and AI. These technologies are tools that help us tell stories and share information like never before. They can be used as incredibly powerful emotional engagement tools enabling us to build empathy and connect with audiences—beyond what TV, computers, or the internet used to offer.

Virtualitics data analytics dashboard in VR/AR – virtualytics.com

This is an exciting time to work with these technologies; most problems are new, and old guidelines often don’t apply. Institutions like MIT are leading the way with radical experiments on new ideas. For example, the MIT Game Lab is creating Play Labs to accelerate startups in AR, VR, and AI. This will undoubtedly inspire many innovators.

The future may be called hyper reality—a combination of AI, AR, VR and MR. One interesting innovation, EscapeLabs, uses augmented reality and mixed reality (AR/MR) to create holographic experiences for escape rooms, team-building exercises, and room-scale puzzle games. Conceivably, the need for clunky headsets and the requirement to connect with a static computer will no longer be necessary. The transition between the virtual and the real world will become easier and quicker—hyper immersive.

The cusp of the 5th industrial revolution – source: CBInsights

The internet existed for many years before we discovered that social sharing was the key to driving engagement. Likewise, it is expected that a combination of these technologies will see similar growth, with job opportunities for designers exploding in the next few years.

New technologies come with many unknowns, creating a need for establishing best practices, standards, and design patterns. It’s assumed that this is why UX designers will have long-term job security in the coming years. There will always be a need for designers employing user-centered design techniques to put warmth into products. The best part about these technologies is that we are still in the beginning stages, and a great deal needs to be figured out by our creative community. Exciting times ahead!

Himanshu Bharadwaj, United States

Himanshu is a digital art-director who’s experienced in crafting UX/UI designs both onsite and remote. In his career, he’s worked for many New York advertising agencies for brands as diverse as Nikon, Samsung, Verizon, Lincoln, American Express, Travelers, Delta, GE, Heineken, IBM, Kraft, Deutsche Bank, Revlon, Sirius, and more. He has worked with small startups doing innovative work and directly with clients through a consultancy practice.