Developer.com content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
In a year where remote working and teleconferencing have dominated business around the world, it is not surprising that Microsoft is focused on improving its collaborative software and working to make communication and interaction even more effective. Microsoft Mesh is a new push from Microsoft that focuses on Virtual Reality, with ties to Microsoft Azure and Microsoft Teams.
In the past, Microsoft has talked highly of Mixed Reality. According to Microsoft, hundreds of millions of people have already experienced Mixed Reality. Over 50% of Fortune 500 companies have purchased their Hololens 2, which Microsoft released last year. With the pandemic, they have seen numerous people pushing harder to find ways to effectively work and collaborate both inside and outside of an organization.
Microsoft Mesh is Microsoft’s newest platform for Mixed Reality. It focuses on creating Mixed Reality solutions that bring together digital and physical worlds. Mix Reality includes augmented and Virtual Reality, as well as any combination of the two.
Mixing VR and Remote Work
One area where Mixed Reality has evolved over the past year is around meetings and working remotely. With Microsoft Mesh, the tools and platform are being provided to make it easier to empower frontline and knowledge workers to be able to collaborate in new and interesting ways. This includes being able to collaborate using Virtual Reality, making it feel as if you were in the same room. Building on services, Microsoft Mesh allows for this sharing of augmented and Mixed Reality. More importantly, it allows for Mixed Reality worlds to be combined with traditional experiences. Those people using a standard flat screen interface can see and interact with those in a pure Virtual Reality world. With things like Azure Object Anchors and Azure Remote Rending, Microsoft is expanding on the features that developers will be able to incorporate into their applications and solutions.
Mixed Reality today is being used in a variety of ways. This includes doing immersive meetups, training, concerts, and presentations. It also is being used to bring remote expertise onsite, where an expert can see what the local attendees are seeing and offer support as if they were present onsite. Collaboration efforts, whether working on a whiteboard or doing modular designs, are also happening. Mixed Reality provides in-situ opportunities without being on site.
Challenges to Mixed Reality in Multi-User Applications
There are challenges to building Mixed Reality applications. First, Mixed Reality tends to use high fidelity 3D models. These are not the easiest things for the average developer to create, let alone bring into a Mixed Reality environment. Second, as Mixed Reality is used collaboratively across multiple devices and platforms, it gets harder. Actions and expressions need to be synchronized across geographically distributed sessions, and holograms have to be kept stable across devices and over time. Additionally, representing people in Mixed Reality in a way that is realistic and not goofy requires a lot of resources.
The Microsoft Mesh Platform
Microsoft Mess resides on top of Microsoft Azure and taps into Microsoft Teams collaborative features and more. More specifically, the Microsoft Mesh Platform provides:
- Natively cross platform support for Halolens, Quest, phones, PC, and the Mac
- A development Platform built on Azure
- A set of Mesh enabled applications
The Microsoft Mesh Developer Platform provides a cross platform SDK that can target AR, VR, PCs, phones, and more. It provides a number of tools and a toolkit.
Presence is core to Mixed Reality. Mesh provides tools to help with digital presence regardless of whether you are focused on avatars or holoportation – it provides the tools that stretch between reach (using avatars) and realism (using holoportation). Mesh provides several tools including an avatar rig and customization studio. It employs AI controlled motion models that can connect a user’s avatar to their motions.
Mesh also uses holoportation technology. This allows a user to deliver 360 degree photorealistic renderings (holoportations) when augmented with external sensors. Suddenly, real images of real people can be seen and used within virtual words.
The Mesh Platform also provides mapping and information on spatial maps. Holograms can be mapped and locked into positions within the real world. These can be shared and persisted using the spatial mapping features within the Mesh Developer Platform.
Additionally, there is support for spatially visualizing world scale holograms using intelligent edge and cloud architecture. This rendering can be done with holographic rendering supported by tapping into Cloud-based tools. This allows for nearly any device regardless of its processing power to be able to get higher fidelity experiences. This Mesh Platform supports holograph rendering, which allows for most 3D file formats to natively render in mesh enabled apps.
Multi-user synchronization is also supported. The ability for people to interact and move virtual items is important to realism. Keeping items synchronized in virtual space for all participants requires positioning to be understood so that virtual items can be handed from one person to the next and align with each user correctly. This synchronized motion and expression of the people in the world is what delivers a sense of true engagement.
In addition to worlds and holograms, it is also important for sound to be synchronized and property positioned across all participants in a virtual world. Microsoft Mesh provides spatial audio to help create the sense of being in the same area.
Core Capabilities of the Mesh Platform
Microsoft is not new to the Mixed Reality space. They are also big players in the collaborative space with tools like Microsoft Teams. This article has scratched the surface on some of the capabilities of Microsoft’s Mesh Platform. This includes its ability to work with immersive presence at multiple levels, the ability to work with spatial maps, the ability to do holographic rendering, and support for handling multiuser synchronization across a multitude of platforms and formats. Its foundation is built on the Microsoft Azure cloud services, so it is easily accessible. If you are looking to do remote collaboration or if you are working with Mixed Reality, then Microsoft Mesh is another set of tools and a platform for consideration.
