LLMs are impressive when used through a computer interface, but it turns out they can be even more impressive when combined with a humanoid robot.
Carolina Parada, Head of Robotics at Google DeepMind, has offered a glimpse into the surprising capabilities of their latest robotic models. Her account highlights the unexpected ways in which these robots are learning and adapting to novel situations. One specific instance was particularly noteworthy, leaving even the engineers who developed the system in awe.
Parada said that the robot was able to play games like naughts and crosses, and tiled anagrams. But it had some unexpected functionality as well. “I think what’s most exciting about these models is that, on many occasions, our own researchers were excited and impressed by what it was doing. It was primarily because the way we were testing it was by putting the robot in front of situations that it’s never seen before, so even we didn’t know whether the robot was going to be able to get it right, and in many occasions it did,” she said.
“So a really cool example that where we were all, like, gasped, was when we showed the video where the robot is actually doing a slam dunk.”
Parada elaborated on the circumstances surrounding the slam dunk: “What was cool about that case is that that day we were just having the creative team come and film the robots, and we asked them to bring toys. We didn’t say anything else. They’re like, ‘Just bring toys to play with the robot, and things the robot hasn’t seen before.'”
“They had no idea what the robot was trained on,” she emphasized. “So they actually brought this little basketball hoop that was a little cute toy with a little ball, and they put it in front of the robot. Again, the robot had never seen anything related to basketball. It certainly has never seen this toy, and they asked it to do a slam dunk of the ball, and we were all like, ‘I have no idea if it would work.'”
The result was astounding. “It actually took not even a quarter of a second, and it actually decided to put the ball inside the basketball hoop, and we were all like, ‘That’s amazing!’ And it was just essentially drawing from Gemini’s understanding of what basketball is and what a slam dunk is. Which is a concept that we wouldn’t have thought of teaching it to do. And essentially it did the right motion, so that was a really cool example.”
This anecdote highlights a significant leap in AI capabilities. The robot, without specific training in basketball or slam dunks, was able to extrapolate from its existing knowledge base and execute a complex action. This suggests a level of understanding and problem-solving that goes beyond rote learning, hinting at genuine cognitive abilities.
While the use of LLMs through computers has become mainstream in recent years — ChatGPT is now the fifth most visited website in the world — the integration of these models with robots is less well understood. These models already have a high level of intelligence, and an understanding of how the world works. And when integrated into a humanoid form factor which can navigate the world, they can keep coming up with surprising results in how they’re able to mimic activities which were previously solely the domain of humans.