Event cameras, which mimic the retina of the eye, have for decades promised a revolution in robotics: they record changes in illumination continuously and allow robots to react faster than conventional cameras. However, data from such sensors looks like a chaotic stream of points rather than familiar images, and processing it required separate algorithms for each task.
Engineers from the University of Pennsylvania (USA) presented Fast Feature Field, or F3 - a universal computational tool that efficiently converts tens of millions of signals per second into information suitable for various robots. The work was published in the journal Science Robotics.
Conventional cameras record full frames at equal intervals, for example 30 frames per second. Event cameras, like retinal photoreceptors, respond to changes in light and report when an area has become brighter or darker. But, unlike a biological eye, they had no common way to extract meaningful patterns from their own data. Without this, the signals are too noisy and voluminous for direct use.
At first the researchers tried to apply mathematical methods of compressing visual information, but after months of experiments they realized there was no reliable approach. So they decided to train a neural network - a form of artificial intelligence that learns from data - to find a way of summarizing the signals on its own. The key was a prediction task: to anticipate future events from past ones, the network must learn both the structure of the scene and the motion within it. Thus F3 learned to identify patterns without a separate algorithm for each task.
A problem of speed remained. It turned out that roughly 95% of possible positions in the data contain no events - most pixels are silent at any given moment. This made it possible to perform computations only where events actually occur, skipping empty space.
In tests on data from cars, quadruped robots and drones in various conditions - from night streets to crowded rooms - F3-based approaches showed high accuracy. Algorithms with F3 worked roughly two to five times faster than previous ones, and the tool itself processed high-definition data 120 times per second (or 440 times per second at lower VGA resolution).
The authors compare F3 to the JPEG and MPEG compression formats, which simplified the exchange of images between devices: the tool gives event cameras a common language, converting their signals into a format compatible with existing software for conventional cameras. In the future the researchers hope to embed F3 directly into camera electronics so that working with it is as simple as with an ordinary camera. For now this is a laboratory development; a commercial product is not yet in question.