Augmented, mixed, and virtual reality offer great new ways to work with 3D objects, 3D charts, data visualizations and games. Not to mention many uses that we haven\u2019t even thought of. They enable entirely new applications and can make using some existing applications a lot better.<\/p>\n\n\n\n
Take a spreadsheet with a lot of data for example. Displaying the data in a chart helps our brains to very quickly make sense of the data and spot things like trends, ranges and anomalies. If it is a 3D chart on a 2D screen, some things might be hidden from view and it can be hard to see relative sizes. Having that same 3D chart in a 3D environment where you can easily walk around it and look at it from different angles or virtually \u201cgrab\u201d the chart and move it around can bring those benefits to 3D charts.<\/p>\n\n\n\n
AR, MR and VR break out of flat displays and place the human in a world that is not limited to a small, flat area. This can offer some advantages to traditional office applications and even more so for new applications that are simply not feasible with flat displays.<\/p>\n\n\n\n
Moving around<\/strong><\/h3>\n\n\n\nMoving around in extended reality (xR) works quite well already. In this article we will use xR as a generic term for augmented, mixed, and virtual reality. In virtual reality, your body usually stays in one place or it moves around in a limited physical space. (This also helps to avoid bumping into objects in the real world.) The virtual space is limitless. There are several methods for moving around in a virtual environment, such as \u201cteleporting\u201d by pointing at a location where you want to go and triggering a jump. There are even hardware solutions like one that lets you fly like a bird<\/a>.<\/p>\n\n\n\nMoving around in augmented reality is, well, moving around in actual reality. The real world is augmented with extra information or objects. In this app<\/a> for example, you can go on a hike and get information about certain spots along the way<\/a>, projected on a 360\u00ba view of the location.<\/p>\n\n\n\nMixed reality is an enhanced form of augmented reality where your view of the real, physical world is enhanced with virtual objects. If you move your head, the virtual objects stay in the same place in the physical world.<\/p>\n\n\n\n
To understand the difference between augmented and mixed reality, think of the augmented reality of using your phone and pointing your camera at something. Your phone adds objects or information on the screen on top of the camera image. Think of Pok\u00e9mon Go. Mixed reality typically requires a headset that continually augments your view of the real world. Meta Glasses and Microsoft HoloLens<\/a> are two examples. <\/p>\n\n\n\nManipulating objects<\/strong><\/h3>\n\n\n\nIdeally, we would like to be able to touch virtual objects with our hands and realistically \u201cfeel\u201d them. Several xR systems already detect arms and hands and simulate direct manipulation. Some can even apply pressure on your fingers so that you can feel like you are touching an object. Over time this will get more and more realistic.<\/p>\n\n\n\n
Other options are controllers. With controllers you can point, click, rotate, and move in space. Much like a traditional computer mouse except in 3D.<\/p>\n\n\n\n
Voice is another option. You can tell the computer what to do with an object. This too is getting better and better with newer iterations of xR devices.<\/p>\n\n\n\n
Interactions in xR<\/strong><\/h3>\n\n\n\nLet\u2019s look at typical interactions like we are used to on regular computers.<\/p>\n\n\n\n
Imagine trying to work with your laptop without using your mouse or keyboard. (No, it doesn\u2019t have a touchscreen. It is a traditional laptop.)<\/p>\n\n\n\n
Siri? Cortana? Google voice recognition? This may work for a limited set of commands but certainly not for everything you\u2019d like to do. How would you navigate a spreadsheet? Create a slideshow? Draw an image?<\/p>\n\n\n\n
Regular interactions like we are used to with mouse and keyboard are still difficult in xR. Today there is no standard mouse for xR. There is no standard pointer or cursor either. Keyboards floating in xR space are less convenient than writing messages using an old cell phone\u2019s number pad. (Remember T9<\/a>?)<\/p>\n\n\n\nAn extra dimension<\/strong><\/h3>\n\n\n\nA traditional computer monitor is convenient. It is flat, it has a fixed size, and it stands still. It is easy to select something on a screen because it is easy to move a mouse pointer. Touch screens are even more intuitive. They do away with the pointer and let you touch an object on the screen directly. You feel your finger on the screen and you see the result of your action right away.<\/p>\n\n\n\n
In contrast, xR environments are three-dimensional, they can take on any size and objects can move around in space.<\/p>\n\n\n\n
This makes it harder to find things because they may be out of view and it makes it hard to select things with a pointer because you have to move the pointer in 3 dimensions instead of 2.<\/p>\n\n\n\n
Some of today\u2019s interaction methods for xR environments have two types of problems:<\/p>\n\n\n\n
\n- They may be useful but they don\u2019t work well enough.<\/li>\n\n\n\n
- They are the wrong tool for the job.<\/li>\n<\/ol>\n\n\n\n
Let\u2019s look at some examples of these types of problems and how they may be fixed or avoided.<\/p>\n\n\n\n
Great for specific applications<\/strong><\/h3>\n\n\n\nSome interaction methods are suitable in principle but they just don\u2019t work well enough yet. Microsoft\u2019s HoloLens<\/a> has a voice interface that doesn\u2019t always understand what you say and with Meta\u2019s headset it is possible, but still hard to manipulate objects with your hands.<\/p>\n\n\n\nMovies show us what it would be like if everything worked well. The movie \u201cIron Man\u201d<\/a> for example. Jarvis, the AI assistant, understands everything Tony says and all virtual objects can be grabbed and moved by hand.<\/p>\n\n\n\nI\u2019m sure that we will get there sooner rather than later. Others seem to think so too<\/a>. The headsets on the market right now are still regarded as developer models. Several companies are spending a lot of resources on solving these issues as soon as possible.<\/p>\n\n\n\nThe right tool for the job<\/strong><\/h3>\n\n\n\nIf you have a screw, a hammer is not a useful tool. In today\u2019s xR environments we often lack the right interaction tools; that is if our goal was to replace a traditional computer.<\/p>\n\n\n\n
Take voice input for example. You cannot create a drawing using your voice. Other actions like changing the volume of your headset are possible using voice commands but it is slow and imprecise.<\/p>\n\n\n\n
If I say \u201cturn down the volume 10%\u201d, it could be interpreted as: 10% of the total volume, 10% of the current volume or decreasing the volume by 10%. If you had a volume knob instead, you just turn it and hear when it reaches the desired volume \u2013 quick and intuitive.<\/p>\n\n\n\n
User interface elements like knobs, dials, and buttons that are taken from traditional, 2-dimensional user interfaces don\u2019t necessarily translate well into the 3-dimensional xR environment. The constraints of a 2D environment actually make it a lot easier to work with those controls. In xR they would need to be redesigned or replaced by something different.<\/p>\n\n\n\n
We also need better xR user interface implementations that follow the basic usability principles<\/a> that are true in any environment. For example, a consistent way to find and navigate a menu structure, allowing users to easily recover from mistakes, and good ways to give feedback to the user.<\/p>\n\n\n\nDon Norman\u2018s “The design of everyday things”<\/a> is considered a standard text on this subject.<\/p>\n\n\n\nHaving said that, current xR solutions do work really well for certain applications.<\/p>\n\n\n\n
Watching immersive videos (360\u00ba or 180\u00ba) works really well and with cheap tools like Google Cardboard, which quickly transforms your own mobile phone into a VR viewer, a large audience already has access to compelling experiences.<\/p>\n\n\n\n
Take the daily news site Blick.ch for example. We\u2019ve helped them develop apps for VR content<\/a>.<\/p>\n\n\n\n![\"\"](\"https:\/\/augment-it.com\/wp-content\/uploads\/2021\/11\/BlickVR-Promo-Screenshot.jpg\")
<\/figure>\n<\/div>\n\n\nWhat is needed<\/strong><\/h3>\n\n\n\nIdeally, you would want the xR environment to behave and feel like a physical world where you can see, feel, and hear objects and activities, how and where you would expect.<\/p>\n\n\n\n
Currently there are very few systems that offer a fully immersive and realistic experience. The ones that do require powerful and expensive hardware and a dedicated location. The Void<\/span><\/a> is an example of a fully immersive environment with vision, sound, touch, smell, and even temperature.<\/p>\n\n\n\nIf it is not possible to be completely realistic, the best approach is to offer alternatives that work effectively and reliably with the hardware that you have.<\/p>\n\n\n\n
Take virtual reality using a mobile phone and Google Cardboard<\/span><\/a> for example. A primitive solution that simulates a mouse pointer with Cardboard is the \u2018gaze\u2019 method: a pointer \u2013 usually a small dot \u2013 stays in a fixed position in front of your head. If you move your head, the pointer moves with it. If you move the pointer over an object and keep it in the same spot for a short period, the computer knows you want to \u2018click\u2019 in that spot.<\/p>\n\n\n\nThe gaze method may be okay for short interactions like starting a movie but it gets tiresome quickly. It is also totally unsuitable for anything that requires precise movement. Drawing for example.<\/p>\n\n\n\n
With Cardboard you get only one fixed button but no other physical control. This can enhance the gaze method: you can click on an object when the pointer is over it instead of having to wait a few seconds.<\/p>\n\n\n\n
Vive<\/span><\/a> and Oculus<\/span><\/a> have physical controllers that allow much better interaction. You don\u2019t need to move a pointer with your head and eyes anymore; you can just interact with objects using the controllers.<\/p>\n\n\n\n![\"\"](\"https:\/\/augment-it.com\/wp-content\/uploads\/2021\/11\/steamvr-750x400-1.jpg\")
<\/figure>\n<\/div>\n\n\n(Image credit: VR Gamer<\/a>)<\/p>\n\n\n\nThis picture shows a Vive controller that has a laser beam coming out of it in the virtual environment. Point at an object with the laser beam to do something with it.<\/p>\n\n\n\n
Mixed reality devices like Microsoft HoloLens<\/a> and the Meta glasses currently use a mixture of a gaze pointer, hand recognition for gestures, hand recognition for direct manipulation of objects, and voice commands.<\/p>\n\n\n\nWe have found several use cases that work quite well with HoloLens and have developed applications for health, public transport, and tourism, for example.<\/p>\n\n\n\n
This picture shows a full-size model of a ship with a historic representation of what that type of ship looked like many years ago. The overlay is created live, visible to the person who is wearing the HoloLens.<\/p>\n\n\n
\n![\"\"](\"https:\/\/augment-it.com\/wp-content\/uploads\/2021\/11\/AugmentIT-Verkehrshaus-Screenshot-1024x576.jpg\")
<\/figure>\n<\/div>\n\n\nWhere we\u2019re going<\/strong><\/h3>\n\n\n\nMicrosoft is working with other manufacturers like Acer<\/a> to bring (HoloLens) mixed reality devices to a broader audience. Apple has announced it is working on AR and started implementing AR and VR technology at operating system level with MacOS 10.13 and iOS 11. Google, Facebook, and several other companies are also putting a lot of resources into research and development in xR.<\/p>\n\n\n\nIt can be expected that the next generations of mixed reality and virtual reality devices will see significant improvements.<\/p>\n\n\n\n
Currently the HoloLens projects its augmented view into a fairly narrow area in front of the wearer\u2019s eyes. The field of view of mixed reality devices will expand.<\/p>\n\n\n\n
Voice recognition will improve so that it becomes more like a truly conversational interface and it can distinguish between different people.<\/p>\n\n\n\n
Eventually, physical controllers will become smaller and better. They may even disappear, as demonstrated in the \u201cIron Man\u201d movie<\/a>. If our hands can be detected with perfect accuracy, we don\u2019t need the controller anymore. This has already happened with touch screens: you don\u2019t need a mouse anymore.<\/p>\n\n\n\nDevices for tactile feedback will enhance the experience so you can feel virtual objects.<\/p>\n\n\n\n
The devices will become faster, smaller, and cheaper. Currently many devices still need to be tethered to a powerful desktop computer but this is only a temporary situation.<\/p>\n\n\n\n
Moving around in augmented reality is, well, moving around in actual reality. The real world is augmented with extra information or objects. In this app<\/a> for example, you can go on a hike and get information about certain spots along the way<\/a>, projected on a 360\u00ba view of the location.<\/p>\n\n\n\n Mixed reality is an enhanced form of augmented reality where your view of the real, physical world is enhanced with virtual objects. If you move your head, the virtual objects stay in the same place in the physical world.<\/p>\n\n\n\n To understand the difference between augmented and mixed reality, think of the augmented reality of using your phone and pointing your camera at something. Your phone adds objects or information on the screen on top of the camera image. Think of Pok\u00e9mon Go. Mixed reality typically requires a headset that continually augments your view of the real world. Meta Glasses and Microsoft HoloLens<\/a> are two examples. <\/p>\n\n\n\n Ideally, we would like to be able to touch virtual objects with our hands and realistically \u201cfeel\u201d them. Several xR systems already detect arms and hands and simulate direct manipulation. Some can even apply pressure on your fingers so that you can feel like you are touching an object. Over time this will get more and more realistic.<\/p>\n\n\n\n Other options are controllers. With controllers you can point, click, rotate, and move in space. Much like a traditional computer mouse except in 3D.<\/p>\n\n\n\n Voice is another option. You can tell the computer what to do with an object. This too is getting better and better with newer iterations of xR devices.<\/p>\n\n\n\n Let\u2019s look at typical interactions like we are used to on regular computers.<\/p>\n\n\n\n Imagine trying to work with your laptop without using your mouse or keyboard. (No, it doesn\u2019t have a touchscreen. It is a traditional laptop.)<\/p>\n\n\n\n Siri? Cortana? Google voice recognition? This may work for a limited set of commands but certainly not for everything you\u2019d like to do. How would you navigate a spreadsheet? Create a slideshow? Draw an image?<\/p>\n\n\n\n Regular interactions like we are used to with mouse and keyboard are still difficult in xR. Today there is no standard mouse for xR. There is no standard pointer or cursor either. Keyboards floating in xR space are less convenient than writing messages using an old cell phone\u2019s number pad. (Remember T9<\/a>?)<\/p>\n\n\n\n A traditional computer monitor is convenient. It is flat, it has a fixed size, and it stands still. It is easy to select something on a screen because it is easy to move a mouse pointer. Touch screens are even more intuitive. They do away with the pointer and let you touch an object on the screen directly. You feel your finger on the screen and you see the result of your action right away.<\/p>\n\n\n\n In contrast, xR environments are three-dimensional, they can take on any size and objects can move around in space.<\/p>\n\n\n\n This makes it harder to find things because they may be out of view and it makes it hard to select things with a pointer because you have to move the pointer in 3 dimensions instead of 2.<\/p>\n\n\n\n Some of today\u2019s interaction methods for xR environments have two types of problems:<\/p>\n\n\n\n Let\u2019s look at some examples of these types of problems and how they may be fixed or avoided.<\/p>\n\n\n\n Some interaction methods are suitable in principle but they just don\u2019t work well enough yet. Microsoft\u2019s HoloLens<\/a> has a voice interface that doesn\u2019t always understand what you say and with Meta\u2019s headset it is possible, but still hard to manipulate objects with your hands.<\/p>\n\n\n\n Movies show us what it would be like if everything worked well. The movie \u201cIron Man\u201d<\/a> for example. Jarvis, the AI assistant, understands everything Tony says and all virtual objects can be grabbed and moved by hand.<\/p>\n\n\n\n I\u2019m sure that we will get there sooner rather than later. Others seem to think so too<\/a>. The headsets on the market right now are still regarded as developer models. Several companies are spending a lot of resources on solving these issues as soon as possible.<\/p>\n\n\n\n If you have a screw, a hammer is not a useful tool. In today\u2019s xR environments we often lack the right interaction tools; that is if our goal was to replace a traditional computer.<\/p>\n\n\n\n Take voice input for example. You cannot create a drawing using your voice. Other actions like changing the volume of your headset are possible using voice commands but it is slow and imprecise.<\/p>\n\n\n\n If I say \u201cturn down the volume 10%\u201d, it could be interpreted as: 10% of the total volume, 10% of the current volume or decreasing the volume by 10%. If you had a volume knob instead, you just turn it and hear when it reaches the desired volume \u2013 quick and intuitive.<\/p>\n\n\n\n User interface elements like knobs, dials, and buttons that are taken from traditional, 2-dimensional user interfaces don\u2019t necessarily translate well into the 3-dimensional xR environment. The constraints of a 2D environment actually make it a lot easier to work with those controls. In xR they would need to be redesigned or replaced by something different.<\/p>\n\n\n\n We also need better xR user interface implementations that follow the basic usability principles<\/a> that are true in any environment. For example, a consistent way to find and navigate a menu structure, allowing users to easily recover from mistakes, and good ways to give feedback to the user.<\/p>\n\n\n\n Don Norman\u2018s “The design of everyday things”<\/a> is considered a standard text on this subject.<\/p>\n\n\n\n Having said that, current xR solutions do work really well for certain applications.<\/p>\n\n\n\n Watching immersive videos (360\u00ba or 180\u00ba) works really well and with cheap tools like Google Cardboard, which quickly transforms your own mobile phone into a VR viewer, a large audience already has access to compelling experiences.<\/p>\n\n\n\n Take the daily news site Blick.ch for example. We\u2019ve helped them develop apps for VR content<\/a>.<\/p>\n\n\n Ideally, you would want the xR environment to behave and feel like a physical world where you can see, feel, and hear objects and activities, how and where you would expect.<\/p>\n\n\n\n Currently there are very few systems that offer a fully immersive and realistic experience. The ones that do require powerful and expensive hardware and a dedicated location. The Void<\/span><\/a> is an example of a fully immersive environment with vision, sound, touch, smell, and even temperature.<\/p>\n\n\n\n If it is not possible to be completely realistic, the best approach is to offer alternatives that work effectively and reliably with the hardware that you have.<\/p>\n\n\n\n Take virtual reality using a mobile phone and Google Cardboard<\/span><\/a> for example. A primitive solution that simulates a mouse pointer with Cardboard is the \u2018gaze\u2019 method: a pointer \u2013 usually a small dot \u2013 stays in a fixed position in front of your head. If you move your head, the pointer moves with it. If you move the pointer over an object and keep it in the same spot for a short period, the computer knows you want to \u2018click\u2019 in that spot.<\/p>\n\n\n\n The gaze method may be okay for short interactions like starting a movie but it gets tiresome quickly. It is also totally unsuitable for anything that requires precise movement. Drawing for example.<\/p>\n\n\n\n With Cardboard you get only one fixed button but no other physical control. This can enhance the gaze method: you can click on an object when the pointer is over it instead of having to wait a few seconds.<\/p>\n\n\n\n Vive<\/span><\/a> and Oculus<\/span><\/a> have physical controllers that allow much better interaction. You don\u2019t need to move a pointer with your head and eyes anymore; you can just interact with objects using the controllers.<\/p>\n\n\n (Image credit: VR Gamer<\/a>)<\/p>\n\n\n\n This picture shows a Vive controller that has a laser beam coming out of it in the virtual environment. Point at an object with the laser beam to do something with it.<\/p>\n\n\n\n Mixed reality devices like Microsoft HoloLens<\/a> and the Meta glasses currently use a mixture of a gaze pointer, hand recognition for gestures, hand recognition for direct manipulation of objects, and voice commands.<\/p>\n\n\n\n We have found several use cases that work quite well with HoloLens and have developed applications for health, public transport, and tourism, for example.<\/p>\n\n\n\n This picture shows a full-size model of a ship with a historic representation of what that type of ship looked like many years ago. The overlay is created live, visible to the person who is wearing the HoloLens.<\/p>\n\n\n Microsoft is working with other manufacturers like Acer<\/a> to bring (HoloLens) mixed reality devices to a broader audience. Apple has announced it is working on AR and started implementing AR and VR technology at operating system level with MacOS 10.13 and iOS 11. Google, Facebook, and several other companies are also putting a lot of resources into research and development in xR.<\/p>\n\n\n\n It can be expected that the next generations of mixed reality and virtual reality devices will see significant improvements.<\/p>\n\n\n\n Currently the HoloLens projects its augmented view into a fairly narrow area in front of the wearer\u2019s eyes. The field of view of mixed reality devices will expand.<\/p>\n\n\n\n Voice recognition will improve so that it becomes more like a truly conversational interface and it can distinguish between different people.<\/p>\n\n\n\n Eventually, physical controllers will become smaller and better. They may even disappear, as demonstrated in the \u201cIron Man\u201d movie<\/a>. If our hands can be detected with perfect accuracy, we don\u2019t need the controller anymore. This has already happened with touch screens: you don\u2019t need a mouse anymore.<\/p>\n\n\n\n Devices for tactile feedback will enhance the experience so you can feel virtual objects.<\/p>\n\n\n\n The devices will become faster, smaller, and cheaper. Currently many devices still need to be tethered to a powerful desktop computer but this is only a temporary situation.<\/p>\n\n\n\nManipulating objects<\/strong><\/h3>\n\n\n\n
Interactions in xR<\/strong><\/h3>\n\n\n\n
An extra dimension<\/strong><\/h3>\n\n\n\n
\n
Great for specific applications<\/strong><\/h3>\n\n\n\n
The right tool for the job<\/strong><\/h3>\n\n\n\n
<\/figure>\n<\/div>\n\n\nWhat is needed<\/strong><\/h3>\n\n\n\n
<\/figure>\n<\/div>\n\n\n<\/figure>\n<\/div>\n\n\nWhere we\u2019re going<\/strong><\/h3>\n\n\n\n