NederlandsNVIDIA releases four AI models that let robots and self-driving cars practise in a simulated world
With four new Cosmos 3 models, researchers and companies can make video of a simulated world that reacts to what a robot or car does. The models are free to download. The next step is to measure where they are reliable enough.
A robot or self-driving car learns by practising a lot, but practising in the real world is slow and expensive, and difficult situations rarely come up. That is why companies also let their systems practise in a simulated world. Such an AI model is called a world model: it produces the images the machine would see and lets that world react to what the machine does.
At the start of November, NVIDIA released four new world models. They belong to Cosmos 3, the company's model family, and all four are further-trained versions of Cosmos 3 Nano. Anyone may download them, and the licence also allows commercial use.
Four models, four tasks
Two models are made for cars, and two mainly for robots.
- Transfer-Auto makes practice material for self-driving cars. It produces images from up to eleven cameras at once and measurements from lidar, a laser that measures the distance to everything around the car. With a single instruction, a drive in daytime becomes a drive in rain at night, with the same road and the same cars.
- Sim-Auto makes such images in small pieces one after another, so that a driving system can react and immediately see the consequences of its choice.
- Sim-Transfer turns the simple images from a physics simulator into realistic-looking video. The simulator calculates how everything moves, and the model provides the appearance.
- Sim-Bimanual predicts what a robot with two arms will see next when it moves. The model can be controlled by an AI program or by a person with joysticks, and works for two types of robot, YAM and AgiBot.
Fast enough to react to
A machine that is practising has to see the consequences of its movement at once, because its next choice depends on it. According to NVIDIA, Sim-Bimanual produces 48 frames per second on one powerful chip, with a delay of 166 milliseconds, at 480p quality. Sim-Auto needs more computing power: 17 frames per second for seven cameras plus lidar, on 16 chips, also at 480p.
The image quality is also better, according to NVIDIA. Sim-Bimanual scores higher than DreamDojo, another model, on two test sets of robot images. Sim-Transfer follows the shapes from the simulator more accurately than the ordinary Cosmos 3 model: the error in the distances in the image is 26.5 per cent smaller. These figures come from NVIDIA itself.
NVIDIA also publishes the steps of its training method. One step trains the model on its own earlier predictions. Otherwise, according to NVIDIA, small errors can build up in later images.
Baidu showed this week what such a world can look like for robotaxis: at the Chinese company, the other road users are AI programs that make their own decisions. NVIDIA's models can be downloaded by anyone, and they are also made for robots.
NVIDIA itself still calls this difficult work. The next step is to measure where the models are reliable enough to train and test robots with, and to improve them where they fall short.
What does this mean for the road to the singularity?
Every machine that works in the real world first has to practise a lot. World models that react to what a machine does can make that practice faster. Because NVIDIA openly shares the models and the training method, others can also build such worlds and compare their results with each other. That can speed up the training and testing of robots before they go into the real world.
What is the technological singularity? Read the explanation →
