Precision in Motion: Real-Time Control for Dexterous Robotic Hands

Controlling human-like robotic hands isn't just hard—it's a high-dimensional, contact-rich challenge. This research brings motion control to the forefront with sampling-based model predictive control (MPC), enabling a tendon-driven biomimetic hand to manipulate objects in real time. By combining a physics-based simulator with a visual-language model that refines task objectives on the fly, the system achieves complex in-hand manipulation—like flipping and catching a ball—with no lengthy training required. The result? Adaptive, precise motion control that mimics human dexterity with machine efficiency.

Transcript

00:00:05 Biomimetic and compliant robotic hands offer  the potential for human-like dexterity,   but controlling them is challenging due to high  dimensionality, complex contact interactions, and   uncertainties in state estimation. Sampling-based  model predictive control using a physics   simulator as the dynamics model is a promising  approach for generating contact-rich behavior.   We present the first successful demonstration  of inhand manipulation on a physical biomimetic   tendon-driven robot hand using sampling based  MPC. While sampling based MPC does not require   lengthy training cycles like reinforcement  learning approaches, it still requires adapting   the task specific objective function. To adapt  the objective function, we integrate a visual   language model with a real-time optimizer. We  provide the VLM with a high-level human language   description of the task and a video of the hand's  current behavior. The VLM gradually adapts the  

00:01:02 objective function, allowing for efficient  behavior generation, with each iteration   taking less than 2 minutes. In our experiments,  the hand achieves an average ball rolling speed   of 0.35 radiant per second successful ball  flips and catching with a 67% success rate.   Our results demonstrate that sampling based  MPC is a promising approach for generating   dexterous manipulation skills on biomimetic  hands without extensive training cycles.