Researchers have designed a wrist-mounted device that continuously tracks the entire human hand in 3D. The bracelet, called FingerTrak, can sense and translate into 3D the many positions of the human hand, including 20 finger joint positions, using three or four miniature, low-resolution thermal cameras that read contours on the wrist. The device could be used in sign language translation, virtual reality, mobile health, human-robot interaction, and other areas.
By looking at wrist contours, the technology could reconstruct in 3D, with keen accuracy, where the wearer’s fingers are. The system reconstructs the full hand posture based on the contours of the wrist. Past wrist-mounted cameras have been considered too bulky and obtrusive for everyday use and most could reconstruct only a few discrete hand gestures.
The FingerTrak lightweight bracelet allows for free movement. Instead of using cameras to directly capture the position of the fingers — the focus of most prior research — FingerTrak uses a combination of thermal imaging and machine learning to virtually reconstruct the hand. The bracelet’s four miniature thermal cameras — each about the size of a pea — snap multiple silhouette images to form an outline of the hand.
A deep neural network then stitches these silhouette images together and reconstructs the virtual hand in 3D. Through this method, researchers were able to capture the entire hand pose, even when the hand is holding an object. The novel technical finding in this work was discovering that the contours of the wrist are enough to accurately predict the entire hand posture.