Jeremy Fishel: Nature's Best Manipulator, Man's Best Controller: Hands | Turn the Lens Ep50

Jeremy Fishel: Nature's Best Manipulator, Man's Best Controller: Hands | Turn the Lens with Jeff Frick Ep50

English Transcript 

© Copyright 2026 Menlo Creek Media, LLC, All Rights Reserved 

INTRODUCTION

Jeff Frick: Hey, welcome back everybody. Jeff Frick here coming to you from Land's End in San Francisco. It's a beautiful day. And the Golden Gate Bridge never, ever gets old.

I'm excited to release the next episode in our collaboration with Humanoids Summit. Humanoids Summit is put on by ALM Ventures and Modar Alaoui and the team. It is with Jeremy Fishel and he is the Principal Scientist for Sanctuary AI.

He had a couple of takes that I wanted to highlight. The first is really thinking deeply about what goes on with hands and really the touch and the tactile and the feedback. He has so many great words, tactile, dexterity, manipulation, feeling, slipping, sliding. So touch and sensing and this feedback loop is pretty important to being able to manipulate things.

The other thing is 'Why a hand?' Why try to pursue this complicated device? He talked about three reasons. One is it's a great device. We like them. It's one of the best devices that we've seen in the natural world. So of course, why wouldn't we want to replicate it?

Another reason that you hear quite often is it's because the human world is designed for humans. And so if you want the machines to interact with tools and doorways and all the things that we've designed basically for our size of being, then it makes sense to kind of replicate that and the hand is very specifically part of that, especially when it comes to dealing with tools and in handling the things that are designed for people.

But then the last thing is he talked about controlling machines with degrees of freedom. And his interesting insight was that the more degrees of freedom that we have to manipulate, the harder it is to learn how to control that device. And his example is comparing say a car to an excavator. It's really hard for us to learn how to manipulate complex multi degrees of freedom devices.

Except this one exception, and that's the hand, because obviously we have millions and millions of years of development and evolution to help us control this amazingly complex device. And so his point is that it is the place where we are the most adept at controlling the most complex devices. So using things like teleoperation and other training methods, degrees of freedom and control, we want our robots to have that same level of control that we have in controlling our own hand.

So really great interview, really interesting guy. A lot of fun. So I think you're going to enjoy this next interview with Jeremy Fishel, the Principal Scientist from Sanctuary AI. Thanks for watching. Thanks for listening on the podcast. See you next time.

INTERVIEW 

Jeff Frick:
Hey welcome back everybody. Jeff Frick here coming to you from the Computer History Museum in Mountain View, California. For the second time I've been to the Humanoids Summit. They were here a year ago. I think in the summer time they were in London, back here again and they just announced they'll be in Tokyo next summer [May 2026]. So it's pretty amazing how fast things are moving.

So we're happy to be back and we've got our next guest here. He's been in the space for a while. He's Jeremy Fishel, the Principal Scientist at Sanctuary AI. Or did we decide to go Principal Researcher?

‍Jeremy Fishel:
Both work, Science, Research. We'll take them both. They go both hand in hand.

‍Jeff:
So you're the hand guy.

‍Jeremy:
Yeah.

‍Jeff:
So let's talk about the hand guy. He had a really interesting conversation about hands. So the context is within, say like vision. You're saying vision's kind of solved. Locomotion. There's a lot of things there but your focus is on the hand and feeling within the hand. Why is it such a hard problem and why is it such an important problem?

‍Jeremy:
It's a great question. So hands, dexterity, tactile sensing is what I've been working on the last almost two decades. It's a challenging problem because the physics.

So vision is largely a passive sense, you observe, you don't really interact. You see the world and you process that information. And even a lot of progress with mobility and moving around, the world that they're interacting with has pretty well understood physics. The ground could kind of be trusted to stay there and not move.

Whereas dexterity and manipulation, you're interacting. You have to understand friction, forces, masses, the energy transfer between the hands and the objects. It's just a tremendously challenging problem to solve in simulation. And it's tremendously difficult to do with the hardware because we need to cram all of these degrees of freedom in such a small space, cover it with tactile sensing that doesn't break. Oh man, it's been a problem.

‍Jeff:
So it's great. So you had a session yesterday with Ed Colgate.

‍Jeremy:
Correct, yeah.

‍Jeff:
Which was really fun. And Ed is also a hand guy. Running a big national thing with like five universities and 200 faculty and 100 PhD students. I loved going through some of the real specifics because when we talk about touch it's kind of a generic term. But you

‍Jeremy:
Yeah.

‍Jeff:
Like force sensing, texture discrimination. And these are the really fun ones. Slip detection. Thermal sensors, contact geometry and the great demo that Ed did yesterday was just literally just manipulating a simple object in your hand to show how that dexterity really is delivered via these amazing things.

‍Jeremy:
Absolutely.

‍Jeff:
‍That we have at the end of our arms. How are you going to replicate that?

‍Jeremy:
Oh, wow. That's, that is the challenge no doubt. So Ed's a good friend. I've known him for almost as long as I've been working in this space too. Nicest guy, incredibly humble. And the work they're doing with the Engineering Research Center [HAND ERC] is really trying to tackle a lot of these problems so that ten years from now we've got the great hardware.

At Sanctuary we're really focused on what we can do a few years from now. So we've got these great hydraulic hands that are quite good with getting the high degrees of freedom and strength. We're working on this very durable tactile sensing. Durability is really critical for this hardware.

We've seen a lot of technologies come out of academic labs that are great research tools but they really lack surviving the real world, what we need for industry or consumer type applications, so hitting the durability specs is tricky. And touch especially is incredibly tricky because you want these compliant surfaces that are soft and supple yet survive for a couple of years. Really tough challenge.

‍Jeff:
You know, it's funny because Ed said yesterday he's like, you know, it's all about the pressure sensors. But then he said, how many pressure sensors do we have in our hands? The answer is zero.

‍Jeremy:
Yeah. That's right.

‍Jeff:
And then, in that same vein as you just said, the importance of the softness of our fingers, which you don't necessarily think about when you see pinchers and some of these other mechanical things.

‍Jeremy:
Yeah.

‍Jeff:
What role does the soft, is it just the cushion in that force kind of distributor? I mean, how does that play? Because that's going to be hard to replicate in a tough industrial environment.

‍Jeremy:
Absolutely, absolutely. And this is something that actually dates back to my PhD thesis about 13, 14 years ago, where we recognized that the mechanical properties like the skin, its friction and its compliance, have just as much to do with what you feel as the sensing modalities. So you can't overlook that.

A lot of people tend to think of tactile sensing as high resolution force or kind of equate it with vision, like what kind of tactile pixels, a phrase that I don't like. They call it taxels. They call them taxels. They merged tactile and pixel together. So they're trying to go from an image to determine what the feel is? Which just isn't at all how touch works. Touch is so dynamic.

The temporal aspect of touch probably more important than the high resolution spatial aspect of touch. So coverage is incredibly important. I'll take coverage over resolution. Nature's invested in covering our bodies in skin. There's a good reason that you don't have any tactile blind spots anywhere on your body. You could feel anywhere you're touched. Certainly the resolution and capabilities change, but that coverage is everything. So that when I bump into something I can detect that information and react to it.

So I feel like a lot of the research in tactile sensing, they're just focusing on the fingertips. And we need to cover the whole robot.

‍Jeff:
Right, right. Well it's interesting, I learned a little bit about collisions last year. You know when robots run into things or run into people. And it can create a real problem. And that's why most industrial robots at factories are behind fences and yellow tape.

‍Jeremy:
Absolutely.

Jeff:
So I mean, how important is that soft tissue factor in getting the information to the robot that it's having a collision before it knocks grandma on her fanny?

Jeremy: Absolutely. I got two great things on that. One of the things Ed and I always talk about is that manipulation is more about negotiating with the world. Whereas a lot of robotics there's not much negotiation, it's position controlled. And if you're in its way, you better hope you're stronger. It's tough luck. Right.

But this compliance is very critical because when you bump into something you want to have some time to react to that information. In physics, if you have two rigid objects collide, the forces go up really fast. And the only tool you have is to slow down. You have to go slower.

And again, going back to nature, if you look at insects, right, these are these little exoskeletons, rigid exoskeleton shells. If you were to scale that up to the size of a person, the mass of that object would go up with the length cubed and the strength, which is largely the cross-sectional area, would go up with the length squared. So what that means is an insect that was our size, if it wanted to bump into things and not break itself, its only thing it can do is move slower to protect its body. And if you had a giant insect moving slow, every other animal would eat it.

So you have to think about why did nature invest in all this squishy skin on the outside which can get cut, infected? I could die from it, right? And really, all the large animals have this soft, squishy skin so that when they bump into something they can react to that information and not damage their own body, ultimately allowing them to move faster. And I think these same principles are going to apply to robots. We care a lot about speed and how quickly we can execute tasks, and we're not going to get there if we don't learn from these principles of covering the robot and all this compliant, squishy stuff.

‍Jeff:
Wow. I mean, it almost begs the question, should skin, because as a human, skin is our largest organ.

‍Jeremy:
Absolutely.

‍Jeff:
And as you mentioned, it regenerates, it gets cut. A lot of bad things happen. It protects us from the crazy environment, air or water or whatever.

‍Jeremy: Yeah.

‍Jeff:
I mean, do you see a future potentially where skin itself, not necessarily the hand, is something that's wrapped over lots of components of the robot?

‍Jeremy:
Oh, absolutely.

‍Jeff:
To bring those sensors and all that anti-collision stuff?

‍Jeremy:
That's essential. We're working on technologies to solve some of these problems, but no doubt in my mind that that's absolutely essential if we really want a general purpose robot that can react to anything that happens to it. That's just critical.

‍Jeff:
It's pretty interesting in terms of the difference between vision and touch. And there was a great demo. You said it's been in the industry a long time. I just saw it for the first time getting ready for this where they had a person, you know, light a match. Most of us learned to light a match before we probably should, when we were troublesome teenagers.

‍Jeremy:
Right.

‍Jeff:
They had these people trying to light a match after anesthetizing their hand.

‍Jeremy:
That's right.

‍Jeff:
So they couldn't feel the match.

‍Jeremy:
‍It's incredibly important.

‍Jeff:
‍And they couldn't do it.

‍Jeremy:
Yeah. Yeah, it's a great, it's a famous video from Roland Johansson who is a former collaborator of ours from back in the 80s I believe their research was done and they essentially applied a nerve block so they couldn't feel with their skin yet they could still feel with their muscles. They could still move their hand just fine. But this lady had a lot of trouble picking up and lighting a match after that was the case.

And what's really interesting about that video is she ultimately does succeed in lighting the match. It just took a long time. And this is like what I think is fundamental to touch. It allows you to speed things up and it makes everything easier.

‍Jeff:
That's a pretty amazing video. So the next big one is dexterity within manipulation. And again, I think a lot of people think of the actual function of grabbing, but in terms of like approach and how you come at it. And then this really evolution in training.

‍Jeremy
Yep.

‍Jeff:
So that you can get N number of, you don't have to run N number of simulations or write it with a real person. Or have some poor teleoperator have to pick it up a kajillion times, so how does training really change the world that you're operating in? Cause you've been at this awhile.

‍Jeremy:
Yeah. So I think dexterity itself is more about being able to react to the environment intelligently. And it does require these multiple fingers and ability to do sort of complex in-hand manipulations like pick something up and spin it around, mostly to speed up work.

Now, why human hands? This is sort of the question that's been asked for a long time. A lot of times there are sort of three reasons. Two of them, I think are so, so. And one of them I think is really good.

We do like to think of it as the best design and it's certainly in nature the human hand is the most capable manipulator, but that doesn't mean it's optimal in a global sense and that we couldn't build something better.

We also like to think that we've built the world for our hands and bodies. So if we want to open door handles and pickup cups, it's a good thing. However, we would gladly change our environments and infrastructure to support our technologies. We do this all the time as a species. You know, we build roads, we build power infrastructure. We'll change the infrastructure to accommodate our technology.

Really the main reason for me on why hands is that people are really good at controlling them, right. So as you try to increase the complexity of a system like we call it, degrees of freedom, how many joints you can move, like a car is a two degree of freedom system. It goes forward, back, left, right, and it takes us a couple of months to train and learn how to get a driver's license.

As you get to like an excavator there's about four degrees of freedom as it spins and moves the shovel at the end. Probably the wrong word for that, but, sure, the internet will correct me. And that takes even more training. And this is a general trend. As we increase the degrees of freedom it gets harder and harder to control.

Except for this one anomaly. Like, we've got our two hands and arms where it's some 50 something degrees of freedom and people are phenomenal at controlling it. And so one of the reasons that I think are critical for hands is if we want to capture human manipulation intelligence, like everything I know about how to interact with the world, we need to have a platform that's easy for us to control.

So I do think teleoperation is very critical in this aspect to learn human manipulation intelligence without the hundreds of thousands of years of evolution required to get there naturally.

‍Jeff:
The other thing that struck me was kind of the not the not touch. The things like the slip and even like they're talking about it in moving, you know slipping and

‍Jeremy:
‍Yeah, yeah.

‍Jeff:
you know everything is not like this perfect static, ut you don't really think about that. And that's got to be super hard to program for.

‍Jeremy:
Absolutely. So much of manipulation and interaction is slipping and sliding and you know, you move your fingers over something or you intentionally pass objects between hands. It's very dynamic and it's a challenging, challenging, challenging thing to, one control and get the right sensory information from touch to people and AI policies to hit that same level of performance we can do.

‍Jeff:
So what's your next big, what's your next big breakthrough for 2026? What do you got in your sights? Sights set on?

‍Jeremy:
So at Sanctuary we're probably going to be announcing pretty soon our next generation Dexterous Hand. It's a hydraulic based hand. It's a hard challenge, a hard problem. We've been working on making it more durable and cost effective. Sort of heads down for about a year or two.

We're going to cover this thing in tactile sensing so that we can get to high performance and really excited to show levels of dexterity that I think haven't been demonstrated yet on a system that is durable and more cost effective.

‍Jeff:
Can you tell us when or is it still top secret?

‍Jeremy:
Still Top Secret but hopefully soon, hopefully soon. We don't like to announce things until we're fully confident that we could back them up.

‍Jeff:
So great. Well congratulations. And we'll look forward to seeing that. I wonder if it's going to have skin or gonna have sensors on the hand? I guess time will tell.

‍Jeremy:
We're working on it.

‍Jeff:
All right. He's Jeremy, I'm Jeff, you're watching Humanoids Summit from Mountain View, California. Thanks for watching. See you next time. Take care.

----

Jeremy Fishel: Nature's Best Manipulator, Man's Best Controller: Hands | Turn the Lens with Jeff Frick Ep50

English Transcript 

© Copyright 2026 Menlo Creek Media, LLC, All Rights Reserved