Developing Deviceless for the HoloLens 2 [Part2]

In my previous blog post, I covered a few concepts for deviceless development on the HoloLens 2. This included explaining the pros and cons of authoring mixed reality applications with a physical device. I explained the Unity Engine’s build procedure that can impact your workflow and with tips and tricks to speed up testing.

In this post I will be talking about the Mixed Reality Toolkit, what it is and the input simulation it provides. I will also cover the spatial awareness system it includes along with how you can take advantage of it in your projects. Lastly, an introduction to the HoloLens 2 emulator, the emulators controls, features and explain what can be achieved using the functionality Microsoft has built into it.

Making use of these tools will not only allow you to iterate on your software solutions quicker, but also aid in testing functionality to help produce higher quality mixed reality applications.

The Mixed Reality Toolkit (MRTK) is a cross-platform toolkit for building Mixed Reality experiences for Virtual Reality (VR) and Augmented Reality (AR). The HoloLens is one of many target devices the MRTK can support.

Included in the toolkit is the ability to simulate many different inputs, including the following:

These are mostly used when testing your application in the editor. The left hand can be brought up by pressing the ‘left shift’ key and moved around with your mouse. The right hand can be activated by pressing ‘spacebar’ and again moved around with your mouse. To lock each hand in place this can be achieved by using the hotkey ‘T’ (left-hand) and ‘Y’ (right-hand). This can be useful in the scenario you wish to simulate a grabbing action. For example, grabbing an object and holding it up in the air. Rotation of each hand can be achieved by holding down ‘left control’ which will change the mouse input to act as the rotation control. Picking things up can be achieved with the left mouse button and lastly moving your hands can be engaged by using the mouse scroll wheel.

These shortcuts and keys can be configured (as of version 2.4.0 running Unity 2019.4) through the Mixed Reality Toolkit’s game object setting screen under Input -> Input Data Providers -> Input Simulation Service. This also gives you the option to customize the sensitivity of the mouse rotation and camera movement along with various other settings to fine tune the input simulation to your needs.

Below is a video example that demonstrates these actions:

Spatial Awareness

The MRTK Toolkit additionally provides another useful tool the spatial awareness system that developers can take advantage of. This system provides real-world environmental awareness in mixed reality applications using a collection of meshes, representing the geometry of the environment. 

This allows the interaction between holograms and the real world enhancing the user experience with augmented reality. A great example of this would be throwing a 3D holographic ball down a flight of stairs. Because of the HoloLens 2's ability to understand the real world environment, the ball will bounce and roll down the steps. The MRTK provides us the ability to emulate the spatial mesh that the HoloLens 2 would usually generate and use this inside the Unity editor.

This screenshot shows the mesh options inside the Unity Editor's MRTK configuration window. If you do not have access to a physical device, you can take advantage of this functionality and import a saved 3D scan area to act as the real HoloLens spatial mesh. Importing an existing environment scan gives you the ability to test your application for that scenario inside the Unity editor without need to be at the target location or have the device with you. 

A real-world scenario in which this could be applied is if you were testing your application for an interstate or international customer. It would not be feasible to travel frequently to perform testing, which is why the ability to import spatial scan data is so valuable. This feature is also available inside the HoloLens 2 emulator which we will cover next.

Finally, I want to talk about the HoloLens 2 Emulator developed by Microsoft. It is built ontop of the Hyper-V platform and integrates into Visual Studio allowing developers to debug and run their UWP applications, directly on their machine in an emulated environment. It provides almost all the functionality of a physical HoloLens 2 device including spatial mapping simulation called Rooms, spatial anchoring, and the developer portal. It can be controlled with your mouse, keyboard, or an Xbox controller allowing you to mimic gestures and actions as if you were using a real device. Like the Mixed Reality Toolkit controls the emulator provides you the option to lock hands in place and rotate through the 6 degrees of freedom.

Below is a diagram showing the controls that can be used as of version 10.0.19041.1117

Spatial Mapping

When it comes to displaying holograms and having them interact with the real-world, spatial mapping plays a key role to enhance the experience. The HoloLens can scan the surrounding environment of the user and create a 3D map, allowing the device to understand and interact with the real world. This enables the ability to have holographic items pinned to walls and utilizing gravity, as mentioned earlier, to see how 3D objects interact with the physical environment.

The above image visualizes this spatial mesh showing you what it looks like and how the HoloLens maps its surroundings. When you are developing an application that needs to use this feature without a device it can an interesting problem to solve. You could mock-up some blocks in Unity to represent the real-world environment, however that can take valuable time if detail is important or the environment is large. Another solution is to utilize the Rooms feature of the emulator, which is similar to what we spoke about in the MRTK toolkit with spatial mapping. Rooms allows you to load in captured spatial data recorded from the HoloLens.

Microsoft have provided some example scenes you can use to test in the emulator if you do not wish to record anything.

• DefaultRoom.xef - A small living room with a TV, coffee table, and two sofas. 
• Bedroom1.xef - A small bedroom with a desk.
• Bedroom2.xef - A bedroom with a queen size bed, dresser, nightstands, and walk-in closet.
• GreatRoom.xef - A large open space great room with living room, dining table, and kitchen.
• LivingRoom.xef - A living room with a fireplace, sofa, armchairs, and a coffee table with a vase.
  

These can be located at:

C:\Program Files (x86)\Windows Kits\10\Microsoft XDE\ (version number eg.10.0.19041.0)\Plugins\Rooms

Device Developer Portal

The HoloLens 2 emulator also gives you access to the device developer portal. This is the control panel that contains everything you need to configure and setup with the device. The device portal in the emulator lets you do tasks just as if it was a physical device. Here are a few of the major options available:

• Install, Remove, Stop Applications running on the emulator.
• View System files on the emulator.
• Monitor Performance of the emulated device.
• View Device Information.
• Gives you access to recordings and spatial data.
  

There are tools available to help make it easier to develop Mixed Reality HoloLens applications. Mastering the functionality of these tool I have talked about will not only help save time in development and testing. But allow you to work independent of a physical device and enable you to create robust applications because of improvements to your quality of development.