SoundWave: Using the Doppler Effect to Sense Gestures for Computer Interaction

Gestures are becoming an increasingly popular means of interacting with electronics and computers, but it is still relatively costly to deploy robust gesture-recognition sensors in existing mobile platforms. Developed by a project team at Microsoft Research, SoundWave is a real-time sensing technique that leverages a speaker and a microphone already embedded in most devices to robustly sense in-air gestures and motion around a device. It is capable of detecting a variety of gestures, and can directly control existing applications without requiring a user to wear any special sensors.

Transcript

00:00:00 gestures are becoming an increasingly popular means of interacting with computers however it is still relatively costly to deploy robust gesture recognition sensors in existing mobile platforms Soundwave is a real-time sensing technique that leverages a speaker and a microphone to robustly sends energy sure's and motion around a device it is capable of detecting a

00:00:22 variety of gestures and can directly control existing applications without requiring the user to wear any special sensors on their body to sense motion this technique uses the Doppler effect which describes the change in frequency as a result of relative motion between the sound source and an observer Soundwave uses existing speakers on a device to produce inaudible tones or

00:00:47 frequency between 18 to 22 kilohertz our system then uses the microphone on the same device to pick up the frequency shifted signal to recognize the gesture here we see the tone produced by Soundwave and the shift in frequency around it as a user moves his hand closer and farther from the PC as the hand moves faster the change in frequency is also greater

00:01:12 variety of gestures can be realized by mentioning characteristics of the sent signals such as speed amplitude direction and time variation by combining direction of the motion and it's variation over time the user in this example can change the direction of scroll by performing a double tap gesture using the number of motion events

00:01:32 generated by Soundwave walking towards or away from the laptop can be uniquely detected in this example it is used to lock a users computer screen as he walks away Soundwave is robust to audible sounds for example music can be playing on the same device as Soundwave is running on without affecting its sensing capabilities this example application also demonstrates the use of two hand

00:01:57 gesture here we see a user playing a game of tetris by performing different gestures slow tapping gestures are used to move blocks left while faster taps towards the right two-handed gestures are used to flip and pulling the handbag in a quick stroke drops the block this example demonstrates the accurate and quick control that Soundwave is capable of

00:02:24 to establish the robustness and reliability of Soundwave we tested it in a busy and noisy environment you