Ask a stranger off the street what Virtual Reality (VR) is or how it works, and most people will have some inclination of what the technology entails; however, ask that same person about Augmented Reality (AR), and the answers are less likely to be easily gained. Maybe someone will talk about the gaming aspect of the technology, or its earliest incarnation in the failed Google Glass.
While the idea of AR has been around for a while, it didn’t garner worldwide attention until the launch of Pokémon Go, the video game that combined smartphone imagery with a real-world scavenger hunt to let gamers experience both real and virtual worlds. But what, exactly, is AR? And as a burgeoning technology, does it offer potential beyond gaming and entertainment as a solution to real-world problems?
This article explores a variety of AR pilot programs and applications across multiple industries, including healthcare, law enforcement, education, logistics, construction, and more. As you’ll see, AR is being transformed into a platform for use across multiple industries for practical uses — a technology that could go from simple pilot projects today to something people use every day as part of their daily lives.
What is Augmented Reality?
Augmented reality — sometimes called mixed reality — is a technology that merges real-world objects or the environment with virtual elements generated by sensory input devices for sound, video, graphics, or GPS data. Unlike virtual reality, which completely replaces the real world with a virtual one, AR operates in real time and is interactive with objects found in the environment, providing an overlaid virtual display over the real one.
While popularized by gaming, AR technology has shown a prowess for bringing an interactive digital world into a person’s perceived real world, where the digital aspect can reveal more information about a real-world object than is seen in reality. Imagine doctors being able to see inside a patient during an examination, rather than by viewing scans and x-rays. Imagine kids learning about molecules by being able to interact with them. Imagine being able to see what maintenance and repairs are needed on a piece of machinery just by looking at it.
Devices that power augmented reality are typically dedicated Head-Mounted Displays (HMDs) or Head-Up Displays (HUDs) like harnesses (such as Microsoft’s HoloLens), helmets, or glasses, and handheld devices such as a smartphone or tablet. Products like Intel’s Recon Jet™ Pro Smart Glasses integrate smartphone capabilities into lightweight glasses, enabling see-what-I-see interfacing with remote Enterprise Resource Planning (ERP) systems. More than just a display module, these glasses enable users to see through the eyes of others at remote locations. Some vendors are branching out using other objects to generate augmented elements such as a projector, table, or lamp, or through a flat panel display itself. With these advances, AR users wouldn’t be hindered by the need for bulky headgear or glasses.
The Next Generation of Healthcare
With the high costs of healthcare not going away any time soon, combined with an aging population in many regions, medical technology is looking to help not just cut the costs for hospitals, providers, and insurance carriers, but also provide better, less invasive techniques toward treatment. Augmented reality has found a home here with the technology being used as a preventive measure to provide healthcare professionals with a means to receive data in a non-traditional way.
Healthcare giant Cigna recently launched a program called BioBall that uses Microsoft HoloLens technology in an interactive game to test for blood pressure and body mass index. Patients hold a light, medium-sized ball in their hands in a one-minute race to capture all the images that flash on a screen in front of them. The BioBall senses a player’s pulse and uses responsive light to connect the game with the player’s heartbeat. Once the game is over, patients receive their health numbers privately in their headsets with suggestions offered to be sent to an associated e-mail address. Cigna says the technology is a way to encourage people to take control of their health by knowing their health numbers, so treatment then can be garnered accordingly.
Over at the University of Maryland’s Augmentarium virtual and augmented reality laboratory, the school is using AR in healthcare to improve how ultrasound is done (Figure 1). Using a Microsoft HoloLens and special software, physicians wearing an AR device can look at both a patient and ultrasound imaging directly in front of them instead of having to look at a bulky screen off to the side, said Barbara Brawn-Cinani, Associate Director for University of Maryland’s Center for Health-Related Informatics and Bioimaging (CHIB).
And ultrasound is only the beginning for AR. “We’re only scratching the surface on surgical applications, and we’re planning for similar interfaces for other imaging modalities, and also field testing to determine how well this kind of tool actually works,” Brawn-Cinani said.
Other companies are working on new platforms that use AR for more complex procedures. Scopis, for example, is developing a tool to give surgeons enhanced vision while performing spinal surgery.
Putting Old Knowledge in New Formats
Meanwhile, AR is opening up new ways to teach kids a variety of subjects they might or might not be interested in learning or, in some cases, help those who have trouble in class catch up with the rest of the students. The University of Helsinki’s AR program helps struggling kids learn science by enabling them to virtually interact with the molecule movement in gases, gravity, sound waves, and airplane wind physics. The university found that AR was enough of a boost for these low-performing children to bridge the gap with other students who were also learning science. It was also shown to help improve the quality of learning in high-performing students.
AR also creates new types of learning by transporting “old knowledge” into a new format. For example, the University of Helsinki says AR for learning became much more appealing because kids had already dabbled with the technology before with Pokémon Go. Similarly, using new technology to teach about the old world is also the idea behind Shifu’s Orboot interactive STEM device (Figure 2). Orboot is an augmented reality educational globe that works with a tablet or smartphone for children to learn about history, animals, monuments, language and arts, weather, and culture by interacting with 3D content.
The information is provided through AR-generated 3D models, voiceovers, and music for a more immersive experience for learning. The goal is to promote active learning and memory retention that would not be possible with just video content, according to Shifu.
The New Boardroom
Projection-based augmented reality is emerging as a new way to cast virtual elements in the real world without the use of bulky headgear or glasses. That’s why this type of AR is becoming a popular alternative for use in the office. Startups Lampix and Lightform are working on projection-based augmented reality for use in the boardroom, retail displays, hospitality, digital signage, and other applications.
Lightform’s device can attach to any projector, similar to a Go Pro, and can instantly generate augmented reality elements (Figure 3). This may be useful for doing work with multiple groups in an office, creating elements for artists in a studio, or presenting new products for display in a retail signage application.
“We think the influx point for this type of technology is five minutes,” said Phil Reyneri, Design Director at Lightform. “If you are going to be in an extended period of time, it makes sense to strap on an AR or VR headset. But if you are doing something that is more of a quick-type experience — such as walking through a store or mall — it is more of a frictionless experience. Also, you can have many people simultaneously experience augmented reality because there is not an expensive headset that an individual has to wear.”
Lampix has developed a lamp that uses a projection module, vision module, and a computer to project augmented reality on a surface that becomes active. Because no headsets or glasses are needed, the lamp-type device could be placed in a boardroom, where collaborative efforts could be worked on both on- and off-site, said Lampix CTO and co-founder Mihai Dumitrescu.
“AR has to be seamlessly integrated into everyday life in order to make sense. When it gets to that point, applications are endless, and it really has the potential to improve and make a lot of our activities more fluent,” Dumitrescu said. “The focus should not be on the technology itself, but much more on the experience. Lampix has a great potential to accomplish this because it is technology that vanishes into the background. You do not need to see the device, it is just the experience that emerges, that is meaningful and useful.”
From Finding Tiny DNA to the Vastness of Space
Meaningful and useful is what companies and think-tanks are working on worldwide. In fact, augmented reality is finding its way into just about every industry and market, at least in a pilot phase. In Germany, for example, FleetBoard is in the development phase of an app that tracks logistics for truck drivers to help with the long series of pre-departure checks before setting off. The FleetBoard Vehicle Lens app (Figure 4) uses a smartphone and software to provide live image recognition to identify the truck’s number plate. Then the relevant information is superimposed in augmented reality, speeding up the pre-departure process.
FleetBoard is also working with Microsoft’s HoloLens to create a new way to manage truck fleets. On an abstract road map, the surface is brought to life with augmented reality, allowing fleet managers to monitor vehicles live in a 3D landscape to prevent accidents and delays, and to warn drivers of possible problems on the road ahead.
Last winter, Delft University of Technology in the Netherlands started working with first responders to use AR as a tool in crime scene investigation. The handheld AR system allows on-scene investigators and remote forensic teams to minimize the potential for contamination. AR also puts multiple sets of eyes on a crime scene that normally would have only the first responder there. This could be extremely helpful in finding traces of DNA, preserving evidence, and getting medical help from an outside source.
Sandia National Laboratories is working with augmented reality as a tool to improve security training for users who are protecting vulnerable areas such as nuclear weapons or nuclear materials. The physical security training helps guide users through real-world examples such as theft or sabotage in order to be better prepared when an event takes place. The training can be done remotely and cheaply using standalone AR headsets.
In Finland, the VTT Technical Research Center recently developed an augmented reality tool for the European Space Agency (ESA) for astronauts to perform real-time equipment monitoring in space. Because these tasks must be carried out without errors and at the right time, AR prepares astronauts with in-depth practice by coordinating the activities with experts in a mixed-reality situation. The tool makes the invisible visible by enabling the visualization of telemetry data from equipment and other systems onboard the space station such as diagnostics and the latest maintenance data, lifecycle, radiation, pressure, or temperature — both in space and on the ground — and displayed on AR glasses.
In the U.S., Daqri International uses computer vision for industrial AR to enable data visualization while working on machinery or in a warehouse. The glasses and headsets from Daqri display — in the field of view — project data, tasks that need to be completed, and potential problems with machinery, or even where an object needs to be placed or repaired. Using AR can improve safety and efficiency in an industrial job space while making repairs, maintenance, and inventory management easier.
Augmented reality merges real-world objects or the environment with virtual elements generated by sensory input devices for sound, video, graphics, or GPS data. AR operates in real time and is interactive with objects found in the environment, providing an overlaid virtual display over the real one.
Although Pokémon Go continues to be popular with gamers, companies around the world are already demonstrating the potential for using AR to solve real-world problems and needs. Industries such as law enforcement, education, logistics, manufacturing, and others are prototyping and using AR technologies, and reporting promising outcomes such as training, monitoring, and collaboration that are more effective and efficient. What’s ahead for AR? With continued development of AR technologies, the reality of widely used practical applications is on the horizon.