How Microsoft HoloLens is Blurring the Line between the Real and the Virtual

Is HoloLens the future of remote work-engagement and much more…?

Did Microsoft’s HoloLens suddenly push us into the age of “Star Trek” and “Minority Report”?

If you have heard about Microsoft HoloLens, maybe you have also come across headlines like these. What is HoloLens? Is it really as revolutionary as it is being touted as? Let me break it down for you.

Source: Link

In this blog, we will take a look at the underlying principles and the construction of HoloLens. Then we will delve into how HoloLens is transforming the corporate and industrial work-culture. Big giants like NASA and Lockheed Martin have already discovered the strength of HoloLens and maybe its the right time for you to make your move.

First, let me tell a bit about myself.

I am Manorama Jha. I started my career in VR one and half years ago in March 2017 as a Research Assistant in IIT Kharagpur and moved to Gridraster Inc. as a Virtual/Augmented Reality Development intern and continued as a full-time Software Development Engineer. My work involves the development and maintenance of proprietary AR/VR/MR solutions on a variety of platforms including Microsoft Hololens and Google DayDream.

Source: Link

This article is organised as follows:

  1. Mixed Reality — An introduction
  2. HoloLens — An Introduction
  3. What’s inside HoloLens?
  4. Capabilities of HoloLens
  5. Applications of HoloLens
  6. A curated list of HoloLens resources

Mixed Reality — An Introduction

Mixed Reality refers to interaction with and manipulation of both physical and virtual objects at the same time in the same frame of reference using natural inputs from the user. Mixed Reality uses next-generation sensing and imaging technologies to blend the real and virtual environments where the user can immerse herself and interact seamlessly with both real and digital (virtual) elements without ever removing the headset.

Source: Link

A short demonstration on Mixed Reality

Mixed Reality is sometimes also termed as Hybrid Reality since it technically merges the results of both real and virtual world.

Source: Link
Source: Link

A bit of History — The first ever immersive mixed reality system was seen in the 1990s. The first MR system Virtual Fixtures platform developed at the U.S. Air Force’s Armstrong Laboratories was developed to provide sight, sound and touch all at the same time. This project successfully demonstrated that human performance could be amplified by the introduction of spatially registered virtual objects overlaid on top of a person’s direct view of a real physical environment.

Types of MR Devices: Over the past several decades human inputs and computer inputs have been surveyed and now this field of computer science is widely termed as HCI (Human-Computer-Interaction). Human give their inputs mostly through well-known input devices like keyboard, mouse, joystick, etc. The recent advancements in the technologies of sensors and data-processing have given birth to a new era where a computer is able to receive input directly from a real environment like capturing the position and motion the person’s head position in the world coordinates, surfaces, boundaries or edges, ambient lighting, environmental sound, object recognition and location. As a result, the interaction between computer and human has become equivalent to environmental understanding or perception.

The following two videos give a demo of Holographic and Immersive MR devices:

  • First: Holographic MR devices
  • Second: Immersive MR devices

Some of the well-known mixed-reality devices in recent times are:

  • Microsoft HoloLens
  • Magic Leap
  • Google Glass
  • Acer Windows Mixed Reality Development Edition

Have a look at some pictorial demonstrations to understand MR better:-

Source: Link
Source: Link
Source: Link
Source: Link

That’s all on Mixed Reality Introduction. Let’s move ahead over to next section.

HoloLens — An Introduction

HoloLens is nothing but a Windows computer with a holographic display built into a headset. Hololens lets a user see, hear, and interact with holograms within an environment such as a living room or an office space.

Source: Link

Microsoft has successfully built this adorable MR headset with great features integrated which is portable and can be used wirelessly. It has a total of 8 sensors on device, high-definition lenses and spatial sound technology to give that immersive, interactive holographic experience to the user.

What actually are Holograms?

HoloLens has the capability of letting you create holograms or objects which are made up of light and sound that appear in the world surrounding you, just as if they were real objects. Holograms are equipped with Spatial Understanding or Scene Understanding and have an interface that responds to your gaze, hand-gestures and voice commands and can interact with real-world surfaces around you.

Source: Link

At first, the sensors and cameras on the HoloLens scan the real environment and its Spatial Understanding or Scene Understanding software create a model of the surroundings. Next, it allows the user to create the Holograms of the desired objects with hand gestures and place it anywhere in the environment and start playing with them.

For more details on Holograms please refer to:

HoloLens Hardware

Let me describe the hardware configuration of HoloLens that makes it surprisingly elegant in all dimensions. HoloLens was designed with the following goals in mind:

  • Comfort
  • Power and grace
  • Built-in speakers
  • Spatial sound
  • Sensor fusion
  • The custom holographic processing unit
  • Advanced optics

In case you wanna know about HoloLens hardware in detail, check out the link below:

  • The most eye-catching property of HoloLens is that it requires no wires or phones to get it working — it’s a standalone device.
  • The stars of the show are the holographic high-definition lenses that use a projection system to create multi-dimensional full-colour images with low latency. There are also a host of advanced sensors, like ambient light and four environment sensing cameras that can work together to figure out what you’re doing and what environment you’re in. All of this information is then processed by the custom HPU (holographic processing unit) mapping everything out in real time.
Source: Link
  • There are no fans to keep the headset cool, but there is a vent that allows all that heat to escape from. All of this is backed up by 2GB of RAM and 64GB of onboard storage. As for connectivity, it supports both Bluetooth and Wi-Fi. We also know that the processor uses an Intel 32-bit architecture and an unspecified GPU.
  • If you need to take photos or record mixed reality video, there’s a 2-megapixel HD camera with audio capture, which uses an array of four microphones. Oh, and there are also speakers just above the ears. They use spatial audio, and Microsoft says they synthesize the audio so you can hear your holograms from anywhere in the room.
Source: link

The depth camera

The depth camera on HoloLens is a smaller version of the Kinect camera on Xbox,. It uses one-tenth of the power and has a field of view equivalent to a 15-inch screen from two feet away. The depth camera is what HoloLens uses to “see” and understand your environment. So they can figure out where your desk is, which hand you’re holding out for gestures and help track your head movements with help from other sensors on the device.

The lenses

Blending virtual models, environments and ‘holograms’ with reality means that the lenses are transparent, similar to those found in Google Glass as well as rival smart glasses and goggles from Sony and others. There are two — one for each eye — and they are made up of three layers of glass (blue, green and red).

The holograms

This is where HoloLens gets really exciting because what you see through the lenses isn’t a simple transparent version of your smartphone interface, like Google Glass. Microsoft is going full AR with virtual 3D models of objects that can either appear as part of your real surroundings or combine with reality to make up entirely new ones.

The gestures

The HoloLens is controlled by gestures, voice, and a small Clicker controller to make things a little more varied. The only physical controls on the device are the power switch, a volume button and contrast controls for the holographic lenses.

Source: Link

Still, so far it seems Microsoft has got to an acceptable level of reliability, and it’s going further. Microsoft is constantly working on new gestures and ways to interact with your mixed reality. One example is ‘holding’, which would allow anyone wearing the HoloLens to grasp and manipulate objects in a virtual 3D space. It’s kind of like Leap Motion’s modelling and sculpting apps.

The voice controls

Of course, since this is Microsoft, it should be no surprise that its steadily improving voice assistant, Cortana, is coming along for the ride. She’ll be able to guide you through your HoloLens experience, much like she can on Windows PCs. Gestures don’t always make sense when navigating menus or opening apps, so the HoloLens microphones will be able to capture your voice commands.

Source: Link

The audio

A big focus of VR companies such as Oculus VR has been on the audio as this can make the difference between creating a truly immersive environment and leaving wearers cold. Since the HoloLens is AR, not VR, there’s less of an obsession with tricking your mind into believing it’s somewhere it’s not, but the sound is still part of the package.

As we said, the speakers sit above your ears and will deliver spatial audio. This means that in-app or in-game audio will come from different directions based on where you are in relation to the virtual object making the sound similar to real-life.

What sets HoloLens apart from a traditional PC

Some of the key differentiating features of HoloLens are listed below:

  • Interact beyond the screen

The user can use holograms to visualize and fine-tune a complex design. The user can also interact with Holograms to learn new things about the behaviour of the target object. HoloLens enables you to make decisions more confidently, work more effectively, and bring ideas to life before your eyes.

  • Your world as a canvas.

Microsoft HoloLens perceptively maps your room and mixes holograms with the environment around you. As per your convenience, you can pin holograms to locations wherever you want to.

  • A natural way of interaction

You can interact with the system to create and shape holograms with gestures. Communicate with apps using your voice and navigate with a glance.

  • Create what you imagine:

Create your own holograms and share them with others. Use holograms to visualize how something will look in the physical world whether it’s a new piece of furniture or a new design of a rocket engine. HoloStudio will even let you turn your holograms into physical objects with 3D print compatibility.

  • Visualize your work:

Go beyond what a 2D and 3D render can do by working in three dimensions. Pin holograms to physical objects so you can size and scale them in real time.

  • See your world in a new way:

No wires, No Screens, No Buttons, just transparent lenses interacting freely with holograms without losing sight of your world.

Applications of HoloLens

In this section, I will cover some of the real-world applications that could make HoloLens the next must-have piece of technology in future.

Microsoft HoloLens for remote collaboration at workplaces

The specific features that we need to know about sharing with HoloLens:

  1. Shared, remote experiences — Multiple users of the Microsoft HoloLens, regardless of location, can have a shared holographic experience with additional participants logging in from a PC, tablet, or mobile phone.
  2. Customized avatars — Each participant is represented in physical space by a full-sized avatar that is uniquely differentiated, with support for gaze, gesture, and voice interaction.
  3. Display of 3D models — Import and display 3D content at any size, from tabletop to full scale for shared review and discussion, with the ability to navigate to points of interest in full-scale models.
  4. Annotations — Ability to add text, drawing, and voice annotations that can be attached in 3D space.

Microsoft Remote Assist is a HoloLens app that will allow customers to collaborate securely using heads-up, hands-free video calls from their own separate computers. Users will have the ability to share images and mixed reality annotations, as well as live stream and video capture. Customers can also share what they’re seeing with experts on their Microsoft Teams contact list — this allows them to continue to work the problems at their end while demonstrating what they’re dealing with to an expert.

Meanwhile, Microsoft Layout allows users to design spaces in mixed reality, using a real-world scale. They can import 3D models in order to create room layouts. Users can experience these designs in VR or as high-quality holograms and collaborate in order to edit these designs and see the results in real time. It allows customers to see ideas in the proper context.

Microsoft HoloLens for Industrial Applications

In production environments, having access to live data from IoT-enabled machines for equipment controllers and line supervisors is very useful as they can improve planning processes and maintenance work to the hilt. For manufacturing contractors and field workers, being able to view real-time data in the cloud while doing their regular work means decision-making about production inputs and output volume, among other key metrics, is accurate and timely.

Source: Link

Microsoft HoloLens for Defence and Military

Microsoft’s HoloLens augmented-reality headsets have been used to fight video-game aliens in space, but now they’ve been enlisted for wargames on Earth as well.

Source: Link

Microsoft has already stated it is targeting the business market for the first generation of HoloLens applications, but in a blog post it offers how it might also be used in military engagements:

“By superimposing a digital map onto the actual battlefield, for example, commanders can better understand the terrain on which their troops are moving and more accurately pinpoint where the enemy is keeping its assets. With the enhanced situational awareness that HoloLens provides, both military commanders and soldiers can perform their jobs with greater safety, speed, and effectiveness.”

Source: Link

The HoloLens could also be used to help design and assemble military hardware. It could also be used to help repair and maintain equipment like aeroplanes:

“With HoloLens, the master mechanic can develop a 3D model that overlays instructions on top of the aircraft, showing the mechanic the exact components that need to be moved, replaced, or recalibrated. No longer does the mechanic need to consult the master mechanic or page through manuals to see where specific parts are located. Viewing digital content superimposed on the actual aeroplane, he or she can visualize exactly what needs to be repaired and in what order, and then quickly take action.”

Microsoft HoloLens for Healthcare — Surgery — Medicine

‘HoloLens’ in Healthcare — Hope or Hype?

What if medical students could project bear-sized holographic brains in their dorm rooms when studying for an anatomy exam? What if surgeons could use those holograms to prepare for complex surgeries or even support the operations themselves?

Source: Link
  • Microsoft HoloLens can create a foundation for medical education by allowing students to practice on hologram modelling of human organs and body. According to Dr. Mark Griswold of the Case Western Reserve University, medical students won’t need cadavers for their practice; instead, they can do the same with a holographic human body. With HoloLens, they can not only receive remote instructions on what to do and what not to do but also learn by performing each step of surgery on the holographic model.
  • HoloLens can create a simulated environment for doctors and nurses to get a virtual but realistic experience of dealing with victims during natural calamities like earthquake, drought, tsunami, etc. On the other hand, they can also provide help to people who are facing unnatural disasters like a plane crash or a terrorist attack.
  • Telemedicine can drastically improve as doctors and nurses can experience the situation during disaster scenarios or when the patient is at a distant location by connecting with them via a virtual environment. Clinicians can make telemedicine more interactive and customized as they can enter the patient’s environment and get a more realistic view of the entire situation. HoloLens can also come handy during medical emergencies as doctors can guide any bystander on how to perform certain medical techniques such as CPR to help the patient survive until medical professionals arrive at the emergency spot.
  1. HoloLens In Healthcare — Real World Example 1 — Stryker
  • Stryker, a leading global medical technology company, has set out to improve the process of designing operating rooms for hospitals and surgery centres. Stryker is using HoloLens to build the operating room of the future.
  • Stryker Communications designs state-of-the-art, customized operating rooms using its ByDesign 3D design software. Microsoft HoloLens extends the capabilities of the ByDesign software, bringing their extensive portfolio of digital 3D assets to life and allowing stakeholders to collaborate in ways they couldn’t before.
  • Recognizing that the current model of operating room design needs to be evolved from 2D to 3D, and knowing that the needs of these specialities can be quite divergent, Stryker has found a way to design a shared operating room that can accommodate all surgical disciplines in a far more efficient manner.

2. HoloLens In Healthcare — Real World Example 2 — Case Western Reserve University, the Cleveland Clinic

  • Case Western Reserve and Cleveland Clinic are partnering with Microsoft on HoloLens as part of its collaboration on its Health Education Campus, a state-of-the-art project that will include the schools of nursing, dental medicine, and medicine, including Cleveland Clinic Lerner College of Medicine, a track within Case Western Reserve’s medical school.
  • One of the project’s primary goals is to pioneer new ways of preparing 21st-century health care leaders. Technology is a key element of those efforts, as is an emphasis on inter-professional education. As Cleveland Clinic President and CEO Toby Cosgrove often observe, “health care is a team sport.”
  • The educational opportunities the HoloLens offers extend well beyond health care. Associate Professor of Electrical Engineering and Computer Science Marc Buchner and a few of his students have worked with the device to model molecules or develop gaming applications.

3. HoloLens In Healthcare — Real World Example 3 — University College London

  • University College London is exploring how Microsoft HoloLens can transform medical imaging, like CT scans, into 3D models. Surgeons use these three-dimensional anatomical holograms to plan for complicated surgeries and to help patients get a clearer picture and better understanding of their own bodies.
  • The plan is to use the technology to help surgeons understand the specific anatomy and condition of a patient. When wearing the HoloLens, surgeons will be able to see an image of the patient’s organs.
  • The short-term hope is this will help with pre-surgical planning. The longer-term vision is to super-impose the hologram on the patient during an operation, and for the image to change in real time to reflect what the surgeons are doing.
  • The technology is being created as a collaboration between the trust, University College London Computer Science, and NHS Code4Health, under the platform for enhanced analytics and computational healthcare (PEACH).

Microsoft HoloLens for Architecture and Home Design

A longtime challenge for architects, engineers and construction pros has been communicating an envisioned space with stakeholders before it exists. They’ve gone from vellum drawings and pretty watercolours to 3D computer renderings and tiny foam models — all time-consuming to make an update, and limited in conveying end results. It’s hard to imagine a miniature office as an actual office you’ll work in. It’s tough to envision the scale, size and feel of a 3D image by looking at a 2D screen.

Source: Link

SketchUp Viewer puts Trimble’s popular 3D modelling software onto Microsoft’s self-contained, holographic computer, allowing people to do what Trimble calls “experiential design review.” Users can walk around a tabletop hologram and rescale, move and rotate it. They can also inhabit a 3D model on a 1:1 scale, allowing them to naturally “feel” and understand a model in a physical setting while accessing essential building information modelling (BIM) data. BIM is the digital representation of physical and functional characteristics of a structure.

The technology enables remote, real-time collaboration; faster and fewer iterations; and more confident decisions throughout the entire building process, from design to construction to maintenance. Mixed reality also lets all stakeholders — architects, engineers, construction manager, clients — view and share the same digital assets in the same natural, immersive way.

Source: Link

A curated list of HoloLens resources

In this section, I will am going to list down some of the useful lists of materials for HoloLens.

Development Resources

  1. Fundamentals of Mixed Reality [https://docs.microsoft.com/en-us/windows/mixed-reality/mixed-reality]
  2. Academy of Mixed Reality [https://docs.microsoft.com/en-us/windows/mixed-reality/academy]
  3. Design of Mixed Reality [https://docs.microsoft.com/en-us/windows/mixed-reality/design]
  4. Development of Mixed Reality [https://docs.microsoft.com/en-us/windows/mixed-reality/development]
  5. Open-Source Projects of Mixed Reality [https://docs.microsoft.com/en-us/windows/mixed-reality/open-source-projects]
  6. Case Studies of Mixed Reality [https://docs.microsoft.com/en-us/windows/mixed-reality/case-study-afternows-process-envisioning%2c-prototyping%2c-building]
  7. Resources of Mixed Reality [https://docs.microsoft.com/en-us/windows/mixed-reality/sf-academy-events]
  8. FAQs of HoloLens [https://docs.microsoft.com/en-us/windows/mixed-reality/accounts-on-hololens]

Development Prerequisites

  1. Microsoft HoloLens
  2. A system with Windows 10 installed
  3. Windows 10 Creator Update
  4. PC developer mode enabled
  5. Unity 2018.x and above

Few Official Demonstrative Videos

  1. Microsoft HoloLens: Developer Information [https://www.youtube.com/watch?v=A784OdX8xzI&list=PLZCHH_4VqpRhwzNjMBHJOcfdMhYZzC0K2]
  2. Microsoft HoloLens: Share your Idea [https://www.youtube.com/watch?v=HsnH2Q7HxOA&list=PLZCHH_4VqpRj0Nl46J0LNRkMyBNU4knbL]
  3. Microsoft HoloLens: Transform Your World [https://www.youtube.com/watch?v=qQNYhJH5Sxg&list=PLZCHH_4VqpRhNU8EioE9rvF63L9sNyX0e]
  4. Microsoft HoloLens: Launch Experiences [https://www.youtube.com/watch?v=_3Y7BXEbqcg&list=PLZCHH_4VqpRjjNTKMyrV-hu3iyUvDgou8]
  5. Microsoft HoloLens: Build 2018 [https://www.youtube.com/watch?v=UpmolMrf5HQ&list=PLZCHH_4VqpRiz9os4cJ7UR8BvlaHrEuvx]

Official Github Repos

  1. Holographic Academy [https://github.com/Microsoft/HolographicAcademy]
  2. MixedRealityToolkit-Unity [https://github.com/Microsoft/MixedRealityToolkit-Unity]
  3. MixedRealityToolkit [https://github.com/Microsoft/MixedRealityToolkit]
  4. HoloLensForCV [https://github.com/Microsoft/HoloLensForCV]
  5. MixedRealityCompanionKit [https://github.com/Microsoft/MixedRealityCompanionKit]
  6. MixedRealityDesignLabs_Unity [https://github.com/Microsoft/MixedRealityDesignLabs_Unity]
  7. MixedReality250 [https://github.com/Microsoft/MixedReality250]
  8. MRDesignLabs [https://github.com/Microsoft/MRDesignLabs]
  9. MixedReality-azure-samples [https://github.com/Microsoft/mixedreality-azure-samples]
  10. MixedReality213 [https://github.com/Microsoft/MixedReality213]
  11. reality-augmentation-using-cognitive-services [https://github.com/Microsoft/reality-augmentation-using-cognitive-services]
  12. MRDesignLabs_Unity_Tools [https://github.com/Microsoft/MRDesignLabs_Unity_Tools]
  13. mixed-reality-extension-SDK [https://github.com/Microsoft/mixed-reality-extension-sdk]
  14. Windows-AppConsult-Samples-MixedReality [https://github.com/Microsoft/Windows-AppConsult-Samples-MixedReality]
  15. mixed-reality-extension-SDK-samples [https://github.com/Microsoft/mixed-reality-extension-sdk-samples]
  16. mixed-reality-extension-unity [https://github.com/Microsoft/mixed-reality-extension-unity]
  17. VisualProfiler-Unity [https://github.com/Microsoft/VisualProfiler-Unity]

Last but not least, I would like to convey my thanks and sincere appreciation to all my team-members at GridRaster for providing me with this wonderful opportunity to work and play with this magical device HoloLens which all-together helped me in extending my knowledge about this device and MR Technology.

That’s all about the HoloLens documentary. If you made it till the end, thank you for sticking around for so long 🙂 . Please feel free to leave your comments below for feedback. You can find me on Twitter(@mnrmja007), Facebook(@mnrmja007) and Medium(Manorama Jha).

Software Development Engineer at Gridraster Inc. | Mixed Reality | Augmented Reality | Artificial Intelligence | Computer Vision | www.manoramajha.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store