Ai2 releases open practice world for robots: a robot arm and a table full of blocks

A robot often learns an action by watching examples. Making those examples takes a lot of time: a person controls the robot arm remotely, again and again. And when something then goes wrong, it is often hard to say why. Was it the examples, the model or the situation?

The Allen Institute for AI (Ai2), an American research institute, wants to make this easier with ToyBlocks. It is a simulation: a virtual world on a computer, in which a robot arm moves blocks on a table. The code is open and free to use. ToyBlocks is the first of what Ai2 calls FoundationWorlds, deliberately simple worlds that researchers fully control. They can change the training data, repeat a test and so find out why a robot makes a mistake.

Pushing, stacking and spelling words

The robot arm pushes, turns, stacks and moves the blocks. The sides of the blocks carry letters and pictures. This means the task can also be searching, sorting or arranging, for example building a stack that spells "Ai2". The simulation also changes the size of the blocks, the height of the table and other properties. This shows whether a model really masters the action or has only learned that one situation by heart.

The computer makes the examples itself, with no person at the controls. A planner works out the order of the actions and the movements between them. These plans are then carried out in the simulation and recorded. Hundreds of plans run at the same time on a single graphics card, which makes this up to 24 times faster than in real life. In this way Ai2 is releasing 85,336 long examples: almost 1,600 hours of robot work.

What the first trials show

Ai2 tested three well-known robot models: ManiFlow, pi0.5 and MolmoAct 2. They practised, for example, with blocks on the left side of the table and then had to do the task on the right side. Or they trained on stacks of two and five blocks and were tested on stacks of three and four. According to Ai2, this went badly, even though the models had been trained beforehand on large amounts of data. The team improved the results, but says the problem is far from solved. Measurements like these are exactly what ToyBlocks is meant for.

The release also includes an open model, ManiFlow-TB, which controls the movements of the arm. In the simulation it succeeded in 54 to 59 per cent of 128 trials with three blocks. It was given the plan ready-made. So these figures say how well the model carries out the movements, not whether it can work out for itself what has to happen. The outcome also varies from one test round to the next, because the simulation does not run exactly the same way each time: the same trial can succeed once and fail another time.

From the simulation to a real arm

Ai2 also put models that had trained only in the simulation on a real Franka robot arm, without any examples from the real world. Single actions, such as picking up, putting down and turning over, succeeded there in 81 per cent of the trials, about as often as the 80 per cent in the simulation.

The AI model Molmo 2 also took the role of planner: each time it chose the action and the block, after which ManiFlow carried out the movement. In this way the two spelled the right word with blocks in 15 of 30 trials on the real arm. That was the easiest version, with all letters upright and visible. In the simulation it worked in 55.5 per cent of the trials. Ai2 wants to run more trials to make these figures more solid.

A large AI model can also control the arm. Claude Opus 5.5 from Anthropic received camera images and used commands to point out what the arm should pick up. It solved ordinary stacking puzzles in 30 out of 30 cases. Harder was searching for and spelling three-letter words with up to eight blocks, where in half of the set-ups a letter was missing. The model then had to say that it could not be done. It got this right in 63 per cent of the cases. Notably, the model reached that 63 per cent with descriptions of the actions that it had written itself. With the standard descriptions it was 24 per cent, and with those of a human expert 36 per cent.

According to Ai2, a second version with more data and more actions will follow in a few weeks. Later the institute wants to move on to two arms at the same time and to robot hands that can feel force and touch.

What does this mean for the road to the singularity?

AI can only build factories, move goods and carry out experiments if robots can do that work in the real world. Learning this is still slow, because good examples are scarce and mistakes are hard to explain. ToyBlocks gives researchers a place to measure those mistakes precisely and a large set of examples to learn from.

The first trials show two things. Models that practised only in a simulation can also work on a real arm, although these are still small tests. And the hardest work lies in everything that is slightly different from the training. That is exactly what researchers can now measure and improve, with blocks as the first step towards more complex worlds.

What is the technological singularity? Read the explanation →