The Engineering Reality of Mixed Reality


  1. Breaking down the problem of Mixed Reality.


In the digital age of today, we humans and almost all things we interact with on a daily basis have a digital self in addition to physical existence. Mixed Reality (MR) is the idea of creating an immersive and seamless experience by blending digital entities in a real-world environment. The following video is a demonstration of this idea.

Mixed reality applications can render accurate to-scale digital models of objects — both static and dynamic — and allow the user to interact with them using intuitive gestures that resemble the way they would interact with the real object. High-precision physics simulation allows accurate reproduction of real-world outcomes of these interactions — thus providing a highly capable tool for education, design and planning. Mixed reality is also capable of virtual teleportation of human users by mapping their actions and expressions on virtual avatars, thus taking remote collaboration to a whole new level. Mixed reality tools are set to revolutionize engineering, design, productivity, education and entertainment. This makes mixed reality one of the most actively researched areas of computer science at the moment. To know more about the widespread applications of Mixed Reality, please read my earlier blog. All these amazing capabilities of MR come at a huge computational cost. As a result, deploying MR at scale poses unique engineering challenges. This article breaks down the technical details of Mixed Reality and analyses the computational cost involved in each of the key steps. Mixed Reality owes its recent success hugely to modern computer architectures that involve an optimal workload distribution between the Cloud and the Edge. In this article, we explain the design philosophy of these compute architectures.

Source: Link

Breaking down the problem of Mixed Reality

A typical Mixed Reality pipeline consists of the following components:

Spatial mapping and 3D reconstruction

Spatial mapping is the process of creating a 3D map of the environment that is being scanned by the MR device. This allows the MR device to understand the environment and model interactions with the real world. Spatial mapping is essential for occlusion, collision avoidance, motion planning and realistic blending of the real and virtual worlds.

The three important components that serve as the building blocks of spatial mapping and 3D reconstruction are:

  • Axes: In order to make measurements in the real world and embed virtual objects of the right scale, the MR device must define its global coordinate system. Depending upon the application, the device may use a 2D or a 3D coordinate system. In a 2D coordinate system, the x-axis represents the horizontal direction and y-axis is vertical. On the other hand, in a 3D coordinate system, the x-axis represents the transverse, y-axis, the longitudinal and z-axis, the vertical direction with respect to the initial pose of the user during calibration (that is defined as the origin of the coordinate system).
Source: Link

Scene understanding

Scene understanding refers to the task of analyzing objects in the context of the 3D structure of the scene. This includes the layout of different objects and the spatial, functional, and semantic relationships between objects [cite]. Scene understanding comprises the following steps:

  • Depth estimation: Depth estimation or extraction refers to the task of obtaining a measure of the distance of (ideally) each point of the scene within the current field of view. This is crucial to obtain a representation of the spatial structure of a scene.

3D object rendering

Rendering or image synthesis is the automatic process of generating a photo-realistic or non-photo realistic image or object from a 2D or 3D model (or models — collectively known as a scene file) by means of computer programs. The results of displaying such an image or object are called render. Rendering pipelines in MR typically consist of the following components.

  • Anchoring: Anchors, also referred to as anchor points, are points in the environment that an MR device knows should always hold their respective digital objects (often referred to as abductors). This applies specifically to static digital objects. For example, say you want to place a digital lamp on a table. You would set the anchor to be on top of the table which the device has already discovered and recognized as a horizontal plane. Anchoring is what ensures that a 3D asset remains in place and behaves the way an object actually would if it were resting at that point in physical space even as the MR device changes its position and orientation in space. This is made possible by the ability of the MR device and its virtual camera to keep track of their positions and orientations in space with respect to the anchor points and render the appropriate surfaces and perspectives of the abductor object(s) as they move around the abductor(s) placed via anchor points. Anchor points are the key to creating realistic MR/AR experiences in a convenient way.

Manipulation and Interaction

In the real world, “manipulation” refers to any changes to an object applied by/through the human hand. Similarly, in VR/AR/MR environments, a real hand or an interface tool is used to grasp and move 3D virtual artefacts. This virtual interaction enables a rich set of direct manipulations of these 3D objects. The way a virtual object behaves when touched by human hand is (in most cases) governed by the same laws of Physics as their real-world counterparts. So modelling such interactions depends on our ability to accurately simulate physics within our computational budget. The performance of a computational physics engine depends on the following six factors [source]:

  • The Simulator Paradigm: The computational paradigm of the simulator determines which aspects of the world can be accurately simulated by it. Some of the popular simulator paradigms are constraint-based, impulse-based and penalty based. For a comprehensive overview of simulator paradigms please refer to [Erleben, 2004].

Multimedia gesture recognition

In modern mixed reality applications such as Microsoft Hololens, the user interfaces are based completely on eye-gaze, hand gestures and voice commands. Eye gaze is used to target and track an object and hand gestures and voice are used to grab the object and manipulate/take actions on it. The eye gaze — hand gesture-based UI is oftentimes not as precise as a mouse-keyboard or touch UI in a traditional computer, but the sheer simplicity of putting a HoloLens on and immediately interacting with content is purely magical and allows users to get to work without any other accessories.

Multi-user experience

The power of mixed reality really shines when it comes to shared experiences in the same (possibly remote) environment. Such shared experiences are made possible by co-location of multiple users through virtual teleportation. A similar situation may arise in which a group is distributed across geographies and at least one participant is not in the same physical space as the others (as is often the case with VR). This is known as a remote experience. Often, there can be a blend of these two cases in which a group has both co-located and remote participants (e.g. two groups in different conference rooms).

The following videos demonstrate the two different scenarios:

  • Users in the same physical space
  • Remote: All your users will be in separate physical spaces.

Computational complexity

Having briefly introduced the hierarchy of computational tasks that form the backbone of any Mixed Reality experience, in this section, we analyse the computational workload that each of these tasks poses. This will lead us to identify an optimal workload distribution scheme that conforms with our cost, computational and power constraints.

Mapping and reconstruction

3D surface reconstruction is accomplished by correspondence matching across the digital image sequences. A set of distinctive feature-points are detected in each image. These feature points are searched for correspondence across the rest of the images. Corresponding feature points are then triangulated to obtain their 3D coordinates. Correspondence matching thus results in a dense 3D point cloud of the target object. The worst-case complexity of these algorithms grows quadratically with the number of feature points used.

After 3D point cloud generation, surface reconstruction algorithms are used to fit a watertight surface over those points. The Poisson Surface Reconstruction and Ball-Pivoting algorithm is a popular choice. The computational complexity of Poisson reconstruction is given by its Octree Depth, Solver Divide and Angle Threshold. We will discuss these parameters in brief.

Source: Link
  • Octree Depth: Octree depth is the depth of the octree which is used during the reconstruction process. An octree of depth D produces a three-dimensional mesh of resolution 2D × 2D × 2D. As the octree depth increases, mesh resolution increases exponentially. The effect is increased CPU resource and memory consumption. The following plots from [citation] demonstrate how reconstruction time increases with Octree Depth for different point cloud resolutions.
Source: Link
  • Solver Divide: Solver divide specifies the depth up to which a conjugate gradient solver is used to solve the Poisson equation. Beyond this depth, Gauss-Seidel relaxation is used. If the solver divide is increased, computation time decreases as, at a greater value, Gauss-Seidel relaxation is used instead of gradient solver to solve the equation.
Source: Link
  • Angle threshold: Angle threshold is the value of the maximum allowable angle between the active edge and the new edge created by the rolling ball in the Ball Pivoting algorithm. If the angle exceeds the threshold value, the ball rolling is stopped for that region. This stops the creation of a new edge and the corresponding vertex is omitted. Increase in angle threshold value results in an increase of computation time, as more edges are added to the 3D mesh by the rolling ball. The quality of the reconstructed surface improves with an increase in angle threshold value. Limiting the value of the angle threshold stops the creation of a triangular mesh with high obtuse angles, thus reducing time complexity at the cost of the fidelity of the reconstructed shape. The following figure from [cite] shows the variation of computational time with angle threshold.
Source: Link

Scene understanding

Scene understanding is the process, often in real-time, of perceiving, analyzing and elaborating an interpretation of a 3D dynamic scene observed through a network of sensors. In recent years, Deep Learning Convolutional Neural Networks (CNN) has become the de-facto standard computation framework in computer vision. CNN's can deliver near-human accuracy in most computer vision applications, such as classification, detection and segmentation.

Source: Link

The high accuracy, however, comes at the price of large computational demand. Computational throughput, power consumption and memory efficiency are three important metrics for analyzing the computational burden of deep learning. The computational workload of an algorithm is measured in terms of the number of Floating Point Operations (FLOPs). The following table taken from [reference] presents the computational workload posed by popular deep learning network architectures used in computer vision.

Source: Link

We can observe a general trend of increasing computational workload with increasing accuracy of prediction. As Mixed Reality is highly constrained by the computational resources that are available on the device, the problem thus becomes to pursue the optimal accuracy under a limited computational budget (memory and/or FLOPs). This calls for using lightweight network architectures such as SqueezeNet, MobileNet, ShuffleNet, ShiftNet and FE-Net that offer reasonable accuracy while being low on FLOPs.

The throughput of object detection from a deep neural network running on a certain hardware platform is measured in terms of the number of Frames processed Per Second (FPS). High FPS object detection is crucial for immersive MR experiences. The following table presents the different state of the art deep learning-based object detection algorithms and their FPS numbers while running on a state of the art NVIDIA Titan-X Pascal GPU.

Source: Link

High throughput object detection and instance segmentation require parallel processing hardware like GPUs. The high core count of these processors makes them extremely power-hungry.

The figure below presents the peak performance of recent Nvidia GPUs for single-precision floating-point (FP32) arithmetic measured by GigaFLOPs or GFLOPs and power consumption gauged by Thermal Design Power (TDP).

Source: Link

The following table compares CPU, GPU and FPGA (Field Programmable Gate Arrays) processors in terms of their throughput and power demand for running different deep learning neural network architectures.

Source: Link

The high power requirements along with the size, weight and cost of specialised hardware for deep learning makes it impractical for inclusion in mixed reality setups that are mostly head-mounted and have strict constraints on power consumption, thermal profile, comfort and price. In an age of burgeoning affordable internet connectivity options and with 5G on the horizon, most mixed reality applications have taken to off-loading almost all deep learning computation to cloud-based AI services that offer GPUs, FPGAs and application-specific processors (ASICs) that are optimized for Deep Learning workloads.

Remote Rendering

Rendering or transcoding is extremely processor-intensive tasks that can occupy all available resources. Remote rendering is a name given to the process of offloading rendering and transcoding tasks to another computer. In Mixed Reality, we need to render enormous high-resolution 3D videos. [cite] Instead of clogging on-device compute with this heavy task, the 3D model data can be stored in a secure cloud server, under the control of the content owner, and only 2D rendered images of the models can be streamed back to client requests. This method relies on good network bandwidth between the client and server and may require significant server resources to do the rendering for all client requests, but it is crucial for high-fidelity immersive MR experiences.

Manipulation and interaction

The computational performance of physics simulation depends on the number and complexity of the geometric features present in the CAD model. Even the presence of a single, relatively small geometric feature can increase the size of the underlying discrete physical simulation problem by as much as 10-fold.

Approximations are an effective method of reducing the computational load of the physics simulation. A 2-D model can be dimensionally reduced to a 1-D beam and a 3D solid can be replaced with a 2D surface, which is comparatively computationally less expensive to analyze. In most of the cases, model simplification has little effect on the accuracy of physics simulation provided the objects and their interactions are represented appropriately.

The following diagram from [reference] compares the performance of some leading physics engines for a task of stacking objects:

Multimedia gesture recognition

As discussed earlier, Mixed Reality applications rely on eye gaze, hand gestures and voice commands for immersive human-machine interaction. Reliable detection of these inputs under a plethora of real-world user conditions is a highly challenging task that is being actively researched to date. Human bodies are nonrigid objects with a high degree of variability, scales, orientations, poses, and illumination offer additional factors of variability to the commands.

In order to address this variability, all modern algorithms for these tasks use some form of machine learning and AI. However, AI is computationally expensive and in most cases, the raw user input data may not be transmitted to the cloud for inference because of privacy concerns. Additionally, these inputs must be registered without any noticeable latency and the network bandwidth can act as a bottleneck and ruin the experience by introducing latency. For these reasons, modern mixed reality setups have dedicated on-device chips for low power, low latency AI inference. [cite neural network processors of Apple, Microsoft, Google, Huawei, Samsung, Qualcomm, Movidius, etc.]

Hybrid MR cloud

In the previous section, we discussed the computational workloads presented by each of the tasks that come together to make an immersive Mixed Reality experience possible. MR devices are mostly mobile wearable devices. Hence they must be low power, lightweight and comfortable. Also, the affordability of these devices is key to the widespread adoption of MR. However, some parts of the MR pipeline like scene understanding and rendering require high-power high-performance compute-resources. As high power parallel computing hardware can not be included in a wearable MR headset, running these tasks on the device would introduce high latency and ruin the immersive experience. Off-loading the heavy lifting to cloud servers and streaming the results to the device is a viable solution to this problem. Such a cloud computing service can offer a selection of CPUs, GPUs, FPGAs, and ASICs for each task to choose for minimizing latency and maximizing throughput. This makes these cloud services hybrid.

We must note that not everything can be offloaded and processed in the cloud because of the latency introduced by limited network bandwidth. Hence, jobs that do not require high performance compute are run on the device itself. These jobs especially include those that are the most crucial for generating spontaneous responses to the users’ inputs. The other reason is for processing these operations on the device itself is the privacy concerns about the user data that cannot be uploaded to the cloud.


In this article, we analysed Mixed Reality from a Systems Engineering perspective. We broke the problem into its components and analysed the nature of computational workloads associated with them. Finally, we motivated the design of Hybrid Mixed Reality Cloud as a backbone for supporting scalable and immersive Mixed Reality applications.

Thank you for taking the time to read my article. If you like my work, please consider hitting the clap button and following me on Twitter and Medium.

I would like to thank my wonderful team at GridRaster Inc. for giving me an opportunity to explore this amazing field and participate in cutting edge development.



Software Development Engineer at Gridraster Inc. | Mixed Reality | Augmented Reality | Artificial Intelligence | Computer Vision |

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store