The next time you’re scrolling your phone, take a moment to appreciate the feat: The seemingly mundane act is possible thanks to the coordination of 34 muscles, 27 joints, and over 100 tendons and ligaments in your hand. Indeed, our hands are the most nimble parts of our bodies. Mimicking their many nuanced gestures has been a longstanding challenge in robotics and virtual reality.
Now, MIT engineers have designed an ultrasound wristband that precisely tracks a wearer’s hand movements in real time. The wristband produces ultrasound images of the wrist’s muscles, tendons, and ligaments as the hand moves, and is paired with an artificial intelligence algorithm that continuously translates the images into the corresponding positions of the five fingers and palm.
The researchers can train the wristband to learn a wearer’s hand motions, which the device can communicate in real-time to a robot or a virtual environment.
In demonstrations, the team has shown that a person wearing the wristband can wirelessly control a robotic hand. As the person gestures or points, the robot does the same. In a sort of wireless marionette interaction, the wearer can manipulate the robot to play a simple tune on the piano and shoot a small basketball into a desktop hoop. With the same wristband, a wearer can also manipulate objects on a computer screen, for instance pinching their fingers together to enlarge and minimize a virtual object.
Here is an exclusive Tech Briefs interview, edited for length and clarity, with Xuanhe Zhao, the Uncas and Helen Whitaker Professor of Mechanical Engineering at MIT.
Tech Briefs: What was the biggest technical challenge you faced while developing this ultrasound wristband?
Zhao: Let me give you some background first. Tracking the motion of the hand is extremely important for many applications, especially VR. Another is robotics, especially humanoid robots. In order to teach the humanoid robotic hand to do all kinds of tasks, you need to capture the motion of a human hand.
The challenges for existing technology — the camera is very inconvenient. You need many cameras to capture detailed hand motion. And once your hand is out of the view, you cannot capture the motion anymore.
Data gloves, similarly, cover the hand. They hinder the hand motion and the sensation/touch feeling of the hand. EMG is an interesting technology. However, EMG measures electrical signal for muscle. It can only give you a few discrete hand gestures; it doesn’t give you the continuous hand motions. This is the challenge of the background.
Then we asked the question, “How can we develop it hand-free? We do not want a glove on the camera of this tracking device of very detailed, complicated hand motion. Then we turned to ultrasound, because ultrasound can image the muscles and tendons in the wrist.
Then we discovered that this image of muscles and tendons can be translated to exactly the 22 degrees of freedoms of all joints of the hand. This is a totally different approach from previous ones. We image inside the body using this wristband to take ultrasound image, and then we use AI to translate this image into the 22 degrees of freedoms of the hand.
Then when we have this continuous imaging, we have 30 frames per second. When we have continuous imaging, we can predict continuous motion of the hand with high accuracy. So, this is really the challenge of other technologies and the innovation of ultrasound wristband.
In terms of technological challenges with regards to the wristband, there were a number of them. The number one challenge was how can you predict the 22 degrees of freedom. To really achieve this capability, we developed a new AI algorithm. Based on training data, we can decipher all 22 degrees of freedom.
Then the second challenge was: Each time when you put this wristband on, it will be at a slightly different position of the wrist. It cannot be exactly at the same position. So, how do you make sure there is no variation when you put it on different positions, right? Similarly, we used the AI algorithm to achieve this. I would say these were the two big challenges for the ultrasound wristband that we already addressed.
Tech Briefs: Do you have any set plans for further research, work, etc.?
Zhao: Yes, the work is ongoing in the lab. OK, so I told you about two challenges that we solved. One challenge we have not solved yet is — for this wristband to work, we needed to first train the AI algorithm. So, we needed to first tell the AI algorithm this ultrasound image is corresponding to this hand motion. Then, after training, we have this predictability. That's not very convenient. We really want a wristband that everyone can put on and can use right away.
To do that, we are actually getting data from more users and then developing more universal models that can really predict different people without training their hand gestures. That's one.
The other, of course, we want to make the wristband smaller and last a longer time. These are hardware development; we are working on both. I would say probably on the timescale of a couple of years, we will address these challenges.
Tech Briefs: Those are all the questions I have. Is there anything else you'd like to add that I didn't touch upon?
Zhao: I really want to emphasize this huge impact. For the future of human society, humanized robots will do lots of different work for us. For that work, we need a dexterous robotic hand. How do we train those robotic hands? We need data from real human hands doing all kinds of tasks. I believe this wristband — without the data glove, without the cameras — is a very effective way to obtain training data for future humanized robots. That's one major application.
Another major application is VR. For that we really need hand-gesture prediction, right? So, in that sense, data glove cameras are incorporated as well. And the EMG really has a limited predictability. So, we believe this ultrasound wristband, based on variable imaging, could be the future of really knowing the human hand motions. I really want to emphasize these two future visions and the importance of this ultrasound variable imaging wristband.
Transcript
00:00:00 [MUSIC PLAYING] NARRATOR: Teaching robots to manipulate the world like humans isn't a new idea. For decades, robotics has aimed to replicate the way we grasp, lift, and interact with our environment. But recreating human level dexterity is far more complex than it looks. Today, the question isn't whether robots can grasp and move objects, but how to make them do so with the flexibility, intelligence,
00:00:27 and finesse that humans achieve effortlessly. A team of MIT engineers has designed an ultrasound wristband that precisely tracks a wearer's hand movement in real time. By training the system on an individual's motions, the wristband can learn and interpret hand movements, then communicate them instantly to a robot or virtual environment. The wristband is equipped with an ultrasound sticker the size of a smartwatch and added on-board electronics that are about as small as a cell phone.
00:01:01 Once attached to a human wrist, the device produces ultrasound images of the wrists' muscles, tendons, and ligaments as the hand moves and is paired with an AI algorithm that continuously translates those images into the corresponding positions of the fingers and palm. The team trained the algorithm using carefully annotated ultrasound images, teaching it how different regions correspond to the many degrees of freedom in the human hand.
00:01:30 In demonstrations, the team has shown that a person wearing the wristband can wirelessly control a robotic hand. As the person gestures or points, the robot does the same. With the same wristband, a wearer can also manipulate objects on a computer screen-- for example, pinching their fingers together to enlarge and minimize a virtual object. The researchers feel this could have immediate impact in virtual and augmented reality
00:01:54 and also provide huge amounts of training data for humanoid robots in dexterity tasks, such as performing certain surgical procedures. Next the team plans to further miniaturize the wristband and continue to train its AI on a broader range of gestures and movements from volunteers with diverse hand sizes and shapes. Ultimately, they aim to create a wearable hand tracker anyone can use to wirelessly manipulate humanoid robots or virtual objects with high dexterity control.
