CHANNELS

Aerospace

Automotive

Automated & Autonomous Vehicles

Battery & Electrification Technology

Defense

Drone TV

Electronics

Manufacturing & Prototyping

Materials

Medical

Motion Control

RF & Microwave Technology

Robotics & Automation

Sensors & IoT

Test & Measurement

Aerospace
Automotive
Automated & Autonomous Vehicles
Battery & Electrification Technology
Defense
Drone TV
Electronics
Manufacturing & Prototyping
Materials
Medical
Motion Control
RF & Microwave Technology
Robotics & Automation
Sensors & IoT
Test & Measurement
Aerospace & Defense
Search Results

CARA & HoloLens Give Objects 'Voices' to Help the Blind Navigate

Caltech  researchers have combined augmented reality hardware and computer vision algorithms to develop software that enables objects to "talk." Worn as a portable headset, the technology translates the optical world into plain audio. The team developed a system they call the Cognitive Augmented Reality Assistant (CARA). CARA was developed for Microsoft's HoloLens, a wearable headset that can scan a user's environment and identify individual objects like a table or laptop. With CARA, each object in the environment is given a voice and will "say" its name upon the user's command. This system could be made available in popular public locations like museums and grocery stores to help blind people make their way through unfamiliar spaces.

Topics:
Electronics & Computers Imaging Medical Photonics Software

More From SAE Media Group

    Transcript

    00:00:04 The technology of virtual reality and augmented reality is about acquiring the information about what's out there in the scene and then converting it to other uses. All these things are coming together, largely as a result of consumer technology, not because someone wanted to build a device for the blind. We're just riding that wave and combining these different devices and concepts to make a new thing that never existed before. The hololens creates a 3-D mesh of the physical space to overlay virtual information on top of physical space. What we did is to try to take advantage of that, but overlay sound or auditory objects on top of physical scenes. Imagine you are in a world where all the objects around you have voices.

    00:00:52 When you first walk into a new space, you can simply have the objects call themselves out, one at a time from left to right. "Floor lamp, laptop, picture..." You can switch to Spotlight Mode and you point your head towards different objects in the scene, they get activated and speak their names to you. It tries to communicate the visual world on cognitive level, at the level of language, at the level of understanding. "Navigation started, follow me, follow me." Essentially we invented this virtual guide. It always stays a few steps ahead of the user. It calls out "follow me" up two flights of stairs, around a few corners, a long stretch of corridor and then a final turn into my office. "Wow, that was pretty cool."

    00:01:38 We have seven blind subjects and they all managed to do the task - to get to my second floor office - during their first round. It was very gratifying to see that a blind person could come to the lab, put on our device and make their way through the building without further assistance. The sky's the limit for what kind of functionalities you want to build into a device like this because it's essentially a software problem. We'd like to see this device used and offered at the entrance of large spaces more or less the way you would adopt an audio guide when you walk into a museum. You plug the headphones in, and off you go.