Use highly complex 3D data with Azure Remote Rendering and HoloLens 2 Learn the benefits of the new AR service for your business.
Azure Remote Rendering is a mixed reality service in Microsoft’s Azure Cloud.
High quality, interactive 3D objects are streamed to HoloLens 2 in real time with the highest resolution.
AR, MR and VR headsets have limited computing power: complex 3D models quickly push them to their limits.
A large number of surfaces form a polygon network, or simply mesh, that determines the shape and surface area of an object.
The more dense the information content of a mesh, the more computing power is needed to render the 3D object. This is why designers use a technology called “Mesh Simplification” to optimize complex content for mobile devices. It reduces the number of polygons to a level at which a mobile device can easily display the 3D object. The effort required for this procedure is enormous and can amount to several person-days.
In addition, many important details are lost as a result of the simplification. However, uncompromising visual representation of the details of a 3D object is essential for applications where important and often expensive business and design decisions are made. This is the moment when Azure Remote Rendering comes into play. Microsoft’s streaming service transfers interactive 3D objects with several hundred million polygons to an AR device such as HoloLens 2 – in real time and without noticeable latency.
What are the advantages of Azure Remote Rendering using augmented reality compared to conventional object representation?
Azure Remote Rendering removes all technical barriers. The entire computing power comes from the Microsoft Cloud. Models with complex architecture or design can thus be displayed in full detail on AR-glasses. All you need for Azure Remote Rendering is:
- Your HoloLens 2,
- A network with a Wi-Fi bandwidth of at least 2.4 GHz
- A stable Internet connection with 50 Mbps download and 10 Mbps upload
You can interact with the remotely rendered 3D object any way you like. It can be enlarged and reduced in the room and parts can be colored or switched on and off individually. Changing perspectives by simply walking around the object also allows a much better view of the details. Use “Cut Planes” to create sectional planes in an object that are visible in the polygon mesh. This allows you not only to examine the entire model, but also to inspect the inner workings of the 3D object. The server-side rendering can also be improved to be suitable for your application via debug rendering API.
Which features does Azure Remote Rendering offer?
- Hierarchical state override enables real-time modifications to the appearance of the 3D model.
- Late stage reproduction stabilizes the hologram as you move around.
- Scene lighting provides additional light sources.
- PBR material provides a realistic appearance of the object under different light conditions.
Collaboration with colleagues or customers on a 3D object represented by augmented reality – for example, during product development – ensures that the parties involved have a better understanding. Questions or design suggestions can be discussed with the help of natural pointing and viewing in the room. This means: the better and more detailed the representation, the fewer misunderstandings will occur. This will save both time and money.
The benefits will extend throughout the entire life cycle of a product or object. Even in product development, for instance, when designing a new machine, all versions can be evaluated collaboratively at any time without the need for preparation. Simply load the 3D object into the HoloLens 2 and all the details are ready for interactive inspection using the AR model.
The presentation at the customer’s facility will itself be impressive, thanks to the future-oriented technology of augmented reality. A machine weighing several tons is digitally present and scalable in the meeting room. This gives the customer a better understanding than possible through a sequence of typical PowerPoint slides. In addition, those involved can get a much better idea of the object in its intended place of use – or you can present it in the factory building at the planned location.
How does Azure Remote Rendering work?
Azure Remote Rendering outsources the local computing process to Microsoft Azure Cloud’s high-end processors. There, high-performance Azure GPU virtual machines render the 3D object, code it and send it to the HoloLens 2. Azure Remote Rendering also offers hybrid rendering. Many applications require special user interfaces. With Azure Remote Rendering, the user interface and 3D rendering can be separated. The 3D object comes from the cloud, and the user interface is customized locally for the user. The Azure software merges both components correctly and automatically.
In industrial visualization, there are objects with an enormous amount of detail that are becoming too complex for a single high-end GPU. Azure Remote Rendering solves this problem with multi-GPU rendering in the cloud. The rendering of the object is distributed across several GPU’s and correctly merged again after the rendering.
The Azure Remote Rendering has another core advantage: You can load your 3D objects exactly when they are needed, without an annoying wait. A conversion service optimizes the data in advance and saves it in an Azure Storage account. Supported source formats are FBX (as of version 2011), GLTF (version 2.x) and GLB (version 2.x). This conversion can be automated.
Use cases for Azure Remote Rendering
Azure Remote Rendering in industry: CAD/PLM integration with Augment IT
CAD (Computer-aided design) is a common procedure for digitizing objects. Software such as Autodesk Inventor provides the appropriate tools for designing 3D objects. These include, for example, parametric modelling for particularly intuitive design engineering according to focal points or complete assembly modelling. Such CAD models are then managed in what is known as PLM software.
Put simply, it is product and data management, including product development and network working. All information generated during product development is integrated into such system. For example, PTC offers the well-known Windshill PLM solution. CAD objects or 3D scans (see also Photogrammetry) are often filed in PLM systems for easy access. Augment IT starts here and simplifies the process: The required data can be immediately used in practice via a fast and preferably automated import into Azure Remote Rendering.
This is possible with a special Software Development Kit (SDK). The workflow looks like this:
- Store CAD data in the PLM system
- Upload CAD data to Azure Remote Rendering via direct access to the PLM system
- Seamless representation without loss of quality directly on the device or at the desired location of use
The advantage in terms of time, quality and cost is immense, because the Augment IT SDK, in conjunction with Azure Remote Rendering, reduces several complex work steps to a single, simple process. There is no need to manually prepare the 3D data; instead, the CAD files are automatically uploaded from the PLM object library to the Azure cloud and can be used immediately on site in any scenario without losing detail.
Users call up a catalog function containing the entire object library and history via an intuitive user interface in the front end of the Augment IT software. For example, a mechanical engineer can open a certain object set in a window of the app and simply drag and drop it onto their HoloLens 2. Apart from industrial applications, there are other good use cases that can benefit from a fast interface and Azure Remote Rendering.
Azure Remote Rendering in the health sector
There are numerous applications for augmented reality in medicine. For example, Olympus, manufacturer of medical devices, visualizes its own product range directly in the operating room via AR.
Thanks to the detailed accuracy of Azure Remote Rendering, the use of AR in medicine goes much further. For example, (post-processed) MRT or CT images could be correctly projected onto the patient in perspective. The high level of detail provides the surgeon with exact orientation when planning individual surgical steps distances are spatially precisely represented and surrounding risk factors are highlighted.
In addition to effective cost savings and the relief of the staff involved, one advantage in this area in particular is the fact that minimizing risks is a very important part of surgery, and precise preparation is indispensable. With Augmented Reality and Azure Remote Rendering, the risk of errors is significantly reduced because the preparation can be carried out on a highly detailed and particularly practical 3D model.
Photogrammetry via Azure Remote Rendering in the construction industry
Photogrammetry can be used to create a photorealistic representation of real objects or environments in VR or AR. Many individual photographs of an object from different perspectives are merged into a single 3D model.
This results in an accurate three-dimensional reconstruction of an object, which, however, has a high number of polygons and therefore requires a lot of computing power. With Azure Remote Rendering, photogrammetry scans can be viewed in full resolution with all details. Fast 3D photogrammetric scans thus represent a possible alternative to CAD models that are costly in production.
A good example is its use in the construction industry. Specially equipped drones can fly over and scan construction areas or construction sites. This scanning method saves time because it is independent of ground conditions or construction machinery, and because the image data can be converted into 3D objects. Previous construction progress can now be quickly and directly imported into the Azure Cloud via the Augment IT SDK and evaluated immediately in the planning office.
Azure Remote Rendering and 5G
For smooth, real-time rendering, a network with low latency and sufficient bandwidth is essential. As mentioned earlier, Microsoft recommends approximately 50 Mbps download and 10 Mbps upload speeds for Azure Remote Rendering. Starting in 2020, 5G is being established as the new mobile communications standard in Germany, Austria and Switzerland. At speeds up to 100 times faster than LTE, XR transmissions are becoming possible in real time in mobile communications.
5G will offer features such as Enhanced Mobile Broadband (eMBB) and Ultra-Reliability and Low-Latency Communications (uRLLC). In combination, these technologies enable high data rates with almost no latency (1 ms). This means that the technical requirements of virtual and augmented reality for the mobile network would be met. 5G increases flexibility: As soon as a 5G network is available, your object library can be accessed immediately via HoloLens 2 without WiFi configuration.
Use Azure Remote Rendering in your company
In summary, Augmented Reality in conjunction with Azure Remote Rendering offers the following benefits for your company:
- No loss of quality: Highly detailed, polygon-rich 3D models in AR
- No preparation time: Direct access to each object version at any time in almost any location
- Save time and money: Significantly less personnel and technology expenditure
- Clear picture and lower error rate: Better understanding of the product or object for stakeholders / customers / decision-makers
- Fewer misunderstandings: Collaborative working and improved communication through direct work on the digital, scalable object in the room
- Competitive advantage: Impressive and customer-oriented presentation with future-oriented technologies
Azure Remote Rendering is currently available exclusively for the HoloLens 2 and Windows desktop PCs. We will be happy to help you make profitable use of this groundbreaking technology. Do you have any questions or would you like advice on integrating an Azure Remote Rendering solution into your company and systems?