NederlandsNew software lets robots see at lightning speed with a camera that works like an eye
Researchers at the University of Pennsylvania have made a very fast type of camera easy to use for robots at last. Their software turns its signals into a picture a robot can work with up to 440 times per second, even at night.
An ordinary camera takes picture after picture, for example thirty times per second. For a robot that has to react quickly, that is sometimes too slow: a lot can happen between two pictures. It also goes wrong in the dark or in strong backlight, and fast movements turn blurry.
For years there has been another kind of camera that works like the retina of our eye: a so-called event camera. It does not take complete pictures. Each pixel reports by itself, at that very moment, that the light there is getting brighter or darker. This way the camera sees movement almost instantly, without blurry streaks, even in low light.
The problem: such a camera does not deliver ordinary pictures, but a stream of tens of millions of separate signals per second. Existing robot software cannot do anything with that. For every task, such as estimating distances or seeing what is moving, someone had to build a special calculation method. As a result, these cameras mostly stayed in the lab.
One translation step for everything
Engineers at the University of Pennsylvania (Penn) have found a general solution. Their method is called Fast Feature Field, or F³ for short. The research is published in the journal Science Robotics.
According to the researchers, the breakthrough came when they had a neural network (an AI system that learns from examples) practise one task: predict which signals the camera will give next, based on the signals it has just given. To do that well, the network has to learn what the surroundings look like and what is moving in them, explains first author and doctoral student Richeek Das.
F³ turns what the network learns into a kind of picture with several layers. That is the translation step: after it, software that already exists for ordinary cameras can also work with the event camera, with a small adjustment. A separate method for each task is no longer needed.
Calculating only where something happens
The first version was too slow to steer with directly. The solution came from a simple observation: at about 95 per cent of the places in the picture, nothing happens at a given moment. F³ therefore calculates only where something changes.
This way F³ processes the signals of an HD camera 120 times per second, and at a lower image quality even 440 times per second. Tasks such as estimating distances and working out what is where in the picture run 25 to 75 times per second in HD. According to the researchers, that is about two to five times faster than earlier methods for these cameras, and on several of these tasks it performs best of all known methods.
Tested on a car, a robot dog and a drone
The team tested F³ with recordings from a car, a four-legged robot and a flying platform, by day and by night, indoors and outdoors, in the city and in rough terrain. According to the researchers, F³ can recognise distances and objects at night much faster than an ordinary camera or a laser scanner, even though the system was not specially trained on night-time images.
Together with doctoral students Dexter Ong and Ioannis Asmanis, the team is now building drones that use these cameras to fly on their own through cluttered, poorly lit spaces. The next step is to run F³ inside the camera itself, so that it delivers usable images straight away. The goal is for an event camera to become as easy to use as an ordinary camera, says lead researcher Pratik Chaudhari, associate professor at Penn. Co-author Kostas Daniilidis compares F³ to JPEG for photos: a common language that lets all kinds of software work with these cameras.
What does this mean for the road to the singularity?
Robots that work independently among people need to see quickly and reliably, even when it is dark or something unexpected happens. Event cameras have been able to do this for a long time, but every robot needed its own software for them. F³ turns the camera into an ordinary component that any robot builder can use. Once the method runs inside the camera itself, drones, robots and vehicles can react faster than with ordinary cameras. That brings robots that move safely through our busy, cluttered world another step closer.
What is the technological singularity? Read the explanation →
