NederlandsRobot AI from Seoul first imagines what the result will look like, and only then chooses its movement
Researchers at Seoul National University have built an AI model that lets a robot 'see' in advance how a task will turn out. With those images, the robot chooses its movements. In real tests it completed almost eight out of ten tasks, slightly more often than well-known competitors.
People do it all day without thinking about it: before you pick up a cup, you already see in your mind how your hand will close around it. Robots usually lack that ability. They get an instruction and a camera image, and then directly calculate how they should move. Researchers at Seoul National University in South Korea are now teaching a robot to look ahead first.
Their AI model is called Dynin-Robotics. It was made by the team of Professor Jaeyoung Do, with Hoeun Lee as first author. The research will be presented in December at NeurIPS, one of the largest AI conferences in the world.
First a picture, then the movement
What is special is that one model does everything at once. It understands language, images and video, and it controls the robot's arm. When the robot gets an instruction, the model first creates an image of how the table will look after the next movement, or when the task is finished.
The robot uses that predicted image in two ways. It can serve as a target: this is how it should look, so which movement goes with that? The model can also come up with a few possible movements and then check which one comes closest to the desired end image. You can compare it with a chess player who tries out a few moves in their head before playing one.
The model can also work the other way round. It watches a video of a robot and describes in plain language what that robot has done.
Tested with fruit and blocks
The model learned from about 1.3 million recordings of robots carrying out tasks, from dozens of public collections. The researchers then tested it on a real robot arm. The arm had to move fruit, sort blocks by colour and stack blocks, sometimes in a fixed order of colours.
On average, 78.4 per cent of the tasks succeeded. Well-known models from Nvidia and from the American company Physical Intelligence scored between 73 and 77 per cent in the same tests, according to the researchers. The new model also turned out to be much faster at working out movements than the version the team started with.
Something else stood out in a simulation. The model also understood instructions that do not literally say what has to happen, such as "pick up the flower that stands for love". A version that had learned only movements did this less well. By also learning what the world looks like, the model therefore understands better what is meant.
Next steps
The tests were done in simulations and in prepared set-ups in the lab. The researchers now want to test the model with more objects, tasks and environments. After that they will look at uses in warehouses and factories, and in the longer term at robots that assist surgeons.
What does this mean for the road to the singularity?
Robots that can imagine what is going to happen are an important step towards machines that work independently in the real world. That ability is called a world model: an inner picture of how things move and react to each other. This research shows that such a world model and controlling a robot arm fit well into one AI model, and that they even strengthen each other.
Much still has to happen before such robots work in every home or factory: more kinds of tasks, messier environments and much more practice. But the idea that a robot first 'dreams' how a task will turn out and only then acts brings robots a little closer to the way we ourselves understand the world.
What is the technological singularity? Read the explanation →
