Spinning Up in VR — Part 8: Applications of Virtual Reality

Hi, welcome to Spinning Up in VR! I am Manorama and this is the eighth of a 9-part tutorial on Virtual Reality (VR) for beginners.

In the previous parts, we talked about the technical details and challenges in VR. This chapter gives an overview of the prospective areas where VR is poised to create a revolutionary impact in the near future. If you are interested in reading the previous or the next chapters, the links are at the bottom of this article.

VR in Sports

Let’s have a quick look at VR applications in the area of sports.

• One idea supported by VR is that you can watch a football game from the best seat available in the stadium while enjoying the comfort of not having to leave home. This is normally done with 360-degree video taken from a fixed position. But you can look around and really feel that you are part of it. These kind of experiences are normally not interactive but you can have buttons to press which give you an information overlay or the videos can be captured from a couple seats in different areas of the stadium so you have the freedom to choose where to sit.
• Similar concepts have also been used in theatre so you can appreciate a ballet or an opera show from the front seat without having to pay the premium or being the lucky one — as this view can be shared by everyone. You can even be a part of your favorite rugby team and feel at one with the action. The following are some of the most promising projects bringing VR to sports:

This Prototype World-scale VR Game is the Size of a Football Field

• There has been some hype about using VR for sports training. But various papers have pointed out that training in the real world is still more effective. Apparently in this area what you learn in VR remains in VR. So the skills are not yet transferable. But that doesn’t stop you from having some fun playing some ping pong. In this case, you use the VR controller as your ping pong bat and the ball is animated using physical simulation.

News and Documentary Films

Another very common use of VR is in news and documentary films.

• One of the leading content providers in this format is the New York Times. They have a YouTube channel where you can find a lot of content, offered as 360-degree videos. You can view it on your mobile phone or ideally with a VR Head Mounted Display which blocks out the real world to give you a truly immersive experience. These videos seem to take you to the actual location where the story took place, and you really feel that you’re part of it and that everything there is actually really relevant to you.

• Nonny de la Peña is a leading figure in the area of VR and journalism. She has been using both modal-based VR and 360 videos to reconstruct newsworthy scenarios with an aim that users can experience the news and be part of it. For example in her 2012 piece Hunger in Los Angeles, she used modal-based VR to recreate a factual event where users can join the queue of a food bank, where a man collapsed and had a diabetic seizure due to hunger. It makes users feel like they are really one of the people who were there and that, what happened to this man could easily happen to them one day.
• The BBC has produced a similar piece of VR experience using motor-based VR in which those users will experience the hopes and fears of a terrified Syrian family as they make the perilous journey from Turkey to Greece on smugglers’ boats. Again, It is quite a powerful piece. The immersion supported by VR hardware really plays an important role in making the user feel that they’re one of the refugees.

Scientific Data Visualization

Another application area where VR is really useful is the scientific data visualization.

• Scientific data visualization, as opposed to information visualization, refers to the use of computer graphics for the analysis and representation of simulated or real data. Scientific data is captured or generated from scientific areas such as meteorology, biology, and medicine.
• For instance, a typical example would be a fMRI brain scan. A common feature of scientific data is that they’re normally very high dimensional. In other words, they’re at least 3D so they’re best visualized in 3D rather than 2D. VR enables users to view and interact with this data in the 3D space. This is very useful in terms of education and training as students would be able to really understand and remember data from an immersive and interactive experience more easily than from text description or 2D pictures.

• Scientists could also benefit from being able to visualize and interact with data in VR as a tool to help them to fully comprehend its 3-Dimensional nature. It could also make it a lot easier for doctors to communicate with their patients about their scan results and surgical plans.

For, example Dr Tyler Ard from the University of Southern California gave patients the ability to visualize their own brain using MRI data at the IEEE VR 2017 Conference held in Los Angeles, California. With the help of motion controllers, a physician would be able to navigate around the slice of her patient’s brain and rotate her wrist in any direction to see the brain as a volumetric rendering. So this is a very typical MRI scan. This is just a way to view that type of scan in three dimensions. So this is a more typical slice-based way to view brain data, which you’re viewing it immersively naturally in very high resolution.

Medical Training

Virtual reality can be used in medical training in many different ways.

• First, VR can be used in immersive learning. Medical students can experience a surgical room through a 360-degree video which captures real operations and can be viewed on mobile VR devices. In this case, by being virtually inside the operation room students are not only familiarizing themselves with the medical procedure but also with the emotional stress triggered by an emergency situation that may arise during the operation.

Here is one of the firm successfully carrying out practices in VR:

BioflightVR

• VR could also be used in surgical skill training. The idea here is that before performing surgical procedures on real patients students can try to do it virtually. This is technically a bit more complicated. You would need model based VR, which would need to be programmed to respond to users’ operations with real-time animation. Another issue is that in order for students to really learn how to operate with their hands we would need very high-end VR interaction devices which not only track the position and rotation with high precision but could also provide realistic force feedback. This certainly goes beyond the capacity of any device you can find in a consumer market.

• Finally, VR can help medical doctors to improve their social skills, especially when it comes to dealing with patients who are very demanding very emotional or who have mental health issues that need to be dealt with sensitively. The scenario had to be very interactive in a realistic way. This can implement with model-based VR. Some virtual patients can be animated and programmed to interact with the doctors verbally and non verbally. In this case, the biggest challenge will be to make it autonomous.

Physical Rehabilitation and Psychotherapy

VR has recently made some headline because it helped some paralyzed patients regain some motor control and physical sensation.

• Basically, those patients were asked to visualize moving their legs and they learnt to use their brains signals as a form of input to control their avatar in VR. So they would be working hard on imagining that they were moving their legs while seeing their legs actually move in VR. This then tricked the brain into thinking they had the ability to move their body which re-engaged some spinal cord nerves. The brain signal used here is called EEG, electroencephalography.
• There are other ways to use VR in physical rehabilitation. Most of them follow the same principle which is not the use of VR can really make the rehabilitation task less boring less painful and a bit more fun. We can utilize the 3D tracking ability of the VR controllers to help motor rehabilitation as well. So, for example, patients could be playing ping-pong in VR.

• Another benefit of using VR is that we can quite easily control the level of difficulties in the task. So, maybe when I first started playing ping-pong I could have a massive bat and the ball could be animated to go very slowly. So it’s very easy to hit a ball and I won’t be frustrated. Later when I get the hang of it I can go up a level so I can have a slightly smaller bat, and the ball could move slightly quicker. So this way I can progress in steps, and be in full control of my own rehabilitation which can be easily done in VR. VR has also been used in the area of psychotherapy.

• Noticeably in the treatment of Arachnophobia, the fear of spiders, and Acrophobia the fear of heights. The idea behind this is that by putting someone to the situation they are afraid of but in a risk-free and controlled environment. They can actually gradually learn how to deal with their fear and cope with the situation.
• The same idea has also been applied to treatment for the post-traumatic stress disorder. Where soldiers who have witnessed traumatic events in the war, try to revisit the place in VR but this time without the traumatic event. Gradually and repetitively they retrain their brain not to overreact.

VR in Industrial Training

Let’s consider the industry scenario, start thinking in this way, what if you were able to train newcomers in the industry in virtual reality to give them awareness about the working facility of the industry before they even get into the real terrific environment?

• Earlier this might be supposed to be a fantasy is now very much a possibility due to virtual reality (VR) technology. Now employees can explore the whole detailed 3D environment with realistic 3D models surrounding them and sometimes they can even feel them with haptic gloves which will allow them to gain all of the knowledge and experience they need without the danger or complications of actually having to enter the facility themselves.

• In simple words, VR for industrial training has such a large range of potential because it offers the benefit of exploring a workspace without even having the pain of physically present there. Due to this training sessions can be held without the risk of interfering with natural operations.
• The second most important reason is the engagement the newcomer employees will feel more engaged in the process whereas traditional training programs are supposed to be or can be boring, resulting in the diversion of your candidates. As a result, this will save the time of training the newcomers in reality still we will be benefited with precise results.

VR in Defense

Same as other sectors VR has a great impact on the defence sector too.

• The main advantage of using virtual reality in defence is time and cost. The military training is pretty expensive, especially the airborne training. That’s why it is more cost-effective to use flight simulators for practice than actual aircraft. Additionally, the possibility to introduce any dangerous element into these scenarios makes it a perfect platform for training and practices without causing any actual physical harm to the trainees in practice.
• Generally, the Virtual reality is used for designing combat situations or a ‘virtual war’ for practice and a training tool. The simulated war zone is an ideal way to train soldiers for engagement with the enemy but in a controlled way. It removes the risks associated with a real-world combat situation such as death or injury. The idea behind this is to prepare soldiers for a real, live combat situation. It teaches ways of dealing with unexpected events, for example, a sniper attack, but without putting themselves in danger. It is vitally important that they are taught how to react to dangerous settings where the wrong decision may mean the difference between life and death.

• The Navy has made full use of virtual reality in the form of submarine simulators which mimic the actions of a real submarine. These simulators are mounted onto hydraulic arms which pitch and roll as a real submarine would. This is useful when trying to recreate the actions of a submarine when it dives or ascends. There are other simulators which are static but show changes in instrument readings as part of the simulation. Trainees must react to these changes as if they were onboard a real submarine. They can replicate a real-life emergency which the trainees have to respond to but in a safe, controlled environment.

• A flight simulator is a form of technology most people are familiar with. It contains a cockpit and controls which are seen in real-world aircraft and replicates the movements associated with flying. The pilot controls the simulator by using a joystick which has force feedback thereby allowing him/her to experience how it responds when he/she moves this joystick. A flight simulator is an enclosed unit based on a motion platform which has 6 degrees of movement and responds to changes in weather, turbulence and battlefield conditions. It is mainly used to train pilots but is also used as part of aircraft research and development (R & D). These devices are used to train military pilots for combat missions which include coordination with ground operations, emergency evacuation, e.g. medical and flying whilst under fire.
• The aim is for the pilot to experience a sense of immersion and to feel as if they flying a real aircraft with real controls and under real-life conditions. This means that the simulator must contain the same set up as seen in an actual aircraft, e.g. helicopter and move in a realistic way. If not then the training session breaks down as the pilot suddenly becomes aware that the movements of the simulator are inconsistent with their mental model. This is why it is important that the simulator is programmed in such a way that it mimics the movements of a real aircraft via force feedback which enhances the pilot’s experience.

VR in Education

According to me applications of Virtual Reality in education is most promising.

Let us start with a small demo video demonstration:

• If you managed to watch this video till the end, you might have got the clear idea of how powerful is VR in creating impact in Education in upcoming years.
• Virtual reality is capable in turning the classic studying process into a really fascinating experience for a learner like never before. While learning in a VR environment the classroom is not just limited to four walls, the super-interesting science subjects especially biology don’t lack the ingredients or tools while learning and while learning biology and anatomy the learner can visualize the whole information clearly in front of his eyes and feel it.

Source: Link

• For children who are in early education or initial stage of learning for them it would be great they initialize their learning through experiencing the things around. There’s so much potential for bringing the curriculum to life using virtual reality, from visiting far-flung corners of the world to holding the human heart in your hands.
• The ability to experience training in 360 is invaluable — and imagine budding mechanics viewing a working engine from all angles without leaving the classroom.

Here are few points which can be taken care of to make the belief stronger for how VR can create a huge impact in education:

1. Visualization: The most important and obvious benefit of using VR in education is the advanced way and possibilities of visualizing the environment around you. Virtual reality has become the universal tool to display objects, processes, locations, and historical events in every area.
2. Improvement of education quality: Now as visualization for sure helps to understand and remember the things in the whole different way and due to this students can get even more immersive experiences with training simulations. VR is proven capable of fitting any school and university subject or program.
3. Active participation: As per the recent survey by various organization, these days students of all stages find studying in a conventional classroom a bit boring. Here comes the idea of integrating VR into any learning curriculum is one of the best ways to enhance the learning process for learners of any domain.
4. No language barriers: In educational curriculum, especially for international students the language barrier is one of the common problems. When we choose virtual reality for learning process then the study materials in any foreign language can be integrated into the VR software.

VR in Robotics

Here we go, if you are updated with the latest and hottest trends in robotics and industrial automation then for sure you would have come across terms like teleoperation, telepresence, and teleoperated robots would probably sound very familiar.

Teleoperated robots the term reflects nothing but they are remotely controlled robots. They might have Artificial Intelligence modules but in general, they take their commands from a human operator and learn from them by executing exactly as instructed to reach the higher precise situation of performing the tasks accurately.

In recent times teleoperated robots are mostly used in medical surgeries and military operations.

1. Scenarios similar to critical surgeries are made easier with teleoperated robotic arms or tools because of their ability to reach the tightest places where human hands are incapable to reach and operate.
2. Now in the case of military operations, teleoperated robots are capable of gathering information and perform dangerous tasks like diffusing or moving a targeted explosive.
3. Previously, these teleoperated robots were controlled by a joystick style setup or console-like controllers which were pretty similar to PlayStation, Xbox or Wii consoles. Now, with advanced emerging Virtual Reality and Augmented Reality technologies teleoperated robots are entering a new spectrum — VR and AR controlled teleoperated robots.

Here I will be introducing you all to the two very first eye-catching work which I came across till now.

The very first applications of VR in robotics which I came through was the work described in this paper:

Link to the paper: https://arxiv.org/pdf/1710.04615.pdf

Abstract of the paper: “Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper, we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach to learning visuomotor skills.”

There are two very very important open questions in this paper:

1. Can we build an inexpensive teleoperation system that allows intuitive robotic manipulation and collection of high-quality demonstrations suitable for learning?
2. With high-quality demonstrations, can imitation learning succeed in solving a wide range of challenging manipulation tasks using a practical amount of data?

From the above questions, we get a very clear picture of their contributions. Here is the summary of their contribution:

1. They are building a VR teleoperation system on a real PR2 robot using consumer-grade VR devices(HTC Vive).
2. They proposed a single neural network architecture for all tasks that maps from raw colour and depth pixels to actions, augmented with auxiliary prediction connections to accelerate learning.
3. Overall perhaps their most surprising finding is that for each task approx 30 minutes of demonstration data is sufficient to learn a successful policy with the same hyper-parameters and neural network architecture used across all tasks.

Source: Link

Source: Link

Link to the GitHub: https://github.com/h2r/ImitateLearning-Movo (In case someday someone desire to implement this)

The second applications of VR in robotics which I came through was the work described in this paper:

Let me introduce you all to the second most interesting work. Last year MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) just came up with another awesome work that was a VR-based controller for Teleoperated Robots using an Oculus Rift headset.

Source: link

Link to the paper: https://arxiv.org/pdf/1703.01270.pdf

Abstract: “ Expensive specialized systems have hampered the development of telerobotic systems for manufacturing systems. In this paper, we demonstrate a telerobotic system which can reduce the cost of such a system by leveraging commercial virtual reality(VR) technology and integrating it with existing robotics control software. The system runs on a commercial gaming engine using off the shelf VR hardware. This system can be deployed on multiple network architectures from a wired local network to a wireless network connection over the Internet. The system is based on the homunculus model of mind wherein we embed the user in a virtual reality control room. The control room allows for multiple sensor displays, the dynamic mapping between the user and robot, does not require the production of duals for the robot, or its environment. The control room is mapped to a space inside the robot to provide a sense of colocation within the robot. We compared our system with state of the art automation algorithms for assembly tasks, showing a 100% success rate for our system compared with a 66% success rate for automated systems. We demonstrate that our system can be used for pick and place, assembly, and manufacturing tasks.”

In this above paper these are the following listed contributions:

• They are developing a method for applying the homunculus model of the mind for teleoperation systems to enable user and robot virtual co-location
• They developed a new virtual reality-based telerobotic architecture for control of robotic systems.
• They showcase a series of demonstrations of the telerobotic system with comparisons to current automation techniques.

Source: Link

• The system prototype that MIT’s CSAIL created works by receiving data input from various sensors placed across a room. The sensors generate environment data for better environment understanding which will contribute in better operation of the robots. Wearing a VR headset, the human operator can see through the robot’s eyes and make movements which the robot will duplicate and learn gradually with higher accuracy over time. However, MIT has created two separate models for interacting with the robot — the direct model and the cyber-physical model.
• Let me give a brief on direct model. Here the user sees exactly the same thing what the robot sees through the VR helmet. So, this method provides more precise interaction with the robot because human feels exactly like he is inside the robot himself. However, the interaction between a VR controller and the robot is still significantly high and the lag could often cause nausea for the operator.
• The other model is called the cyber-physical model. The user (human being) works in a way can say almost related to a virtual copy of the robot and it’s an environment where the robot is located. User’s interaction with the virtual copy is replicated by the actual robot. Isn’t it damn interesting even visualize the situation? In this model since the robot and the user is detached so those minor lags won’t affect the human operator health-wise. However, for this model to work accurately more data and a specialized space solely for the cyber-physical model are required.

That’s all about the application in Virtual Reality from my end.

In the later chapter, we will look into the curated list of Virtual Reality resources. Stay tuned!

I hope that anyone who reads the entire series in the given order you will have a clear visualization of VR technology. Please feel free to leave your comments below for feedback. You can find me on Twitter(@mnrmja007), Facebook(@mnrmja007) and Medium(Manorama Jha).

1. A Brief History of Virtual Reality
2. The Modern Concept of Virtual Reality
3. Virtual Reality Hardware
4. Virtual Reality Software
5. Human-Computer Interaction in Virtual Reality
6. Navigation in Virtual Reality
7. Challenges in Virtual Reality
8. Applications of Virtual Reality
9. A curated list of VR resources