Sight for Scale, Sound for Touch: Fusing Internet Video with Acoustic Feedback for Dexterous Robotics

Abstract:

Achieving human-like dexterity and broad real-world generalization remains one of modern robotics' grand challenges. While learning directly from internet-scale, “in-the-wild” human videos offers a powerful way to bootstrap multi-finger trajectories, vision alone remains blind to contact microphysics and often struggles with self-occlusion and missing tactile forces. This talk explores an exciting frontier at this intersection, pairing the macro-level behavioral scale of internet video with the micro-physical grounding of sound. We will examine how we can extract complex dexterous priors from web-scale data, and how vibrotactile and acoustic sensing—using both active skin-conduction tracking and passive slip detection—can provide the real-time, high-frequency feedback needed to stabilize these movements. Rather than relying on vision alone, this work points toward a future where robots learn what to do by watching the web, and how to do it by listening to the world.

Speaker Bio:

Jeff Ichnowski is an assistant professor at Carnegie Mellon University's Robotics Institute and a co-founder of Jacobi Robotics, Inc., a startup that builds software to power the next generation of robotics. His Momentum Robotics Lab explores scalable robot learning algorithms for high-speed, complex motion, task, and manipulation planning, using high-performance computing, optimization, and deep learning.