Imagine: in an emergency, a person asks a helper robot to bring a defibrillator hanging in the corridor. But if in the robot's memory this device is mistakenly recorded as a fire extinguisher, the machine may not find the life-saving device. Such memory errors can negate the benefits of robots in the home and in industry.
Researchers from the Massachusetts Institute of Technology (MIT) have developed a system that helps a robot identify questionable entries in its memory and refine object descriptions. The work is published on the arXiv preprint server and will be presented at the Conference on Neural Information Processing Systems (NeurIPS).
The new method, called UQ-DAAAM (Uncertainty Quantification for Describe Anything, Anywhere, at Any Moment), compares several descriptions of the same object obtained from different viewpoints. This allows the robot to assess how confident it is in what it saw. If confidence is low, the system selects additional angles to reduce uncertainty and combines the descriptions into a more accurate one, which it stores in memory.
The method builds on the earlier DAAAM development, a robot memory system that uses a vision-language model (VLM). Such a model analyzes images and generates text descriptions of objects. The robot stores these captions in a three-dimensional map of the environment and can later refer to them. However, if an object is only partially visible, far away, or in a blurry image, the VLM may make a mistake, for example mistaking a gray pipe on a roof for a squirrel.
UQ-DAAAM adds uncertainty quantification to this scheme, a statistical measure of how much the model doubts its own words. The system calculates the VLM's confidence in the generated description and then checks its consistency with captions for images of the same object from other angles. Both indicators are combined into a single number. If it exceeds a threshold, the description is deemed unreliable and is refined using additional frames.
To select the best angles, the researchers use a heuristic, a simple rule that suggests which frames will reduce uncertainty. The scientists proved mathematically that this heuristic makes it possible to find optimal frames without rerunning the VLM, which sharply reduces the computational load. Thanks to this, a mobile robot can update its memory in real time.
In tests, UQ-DAAAM increased the robot's confidence and the accuracy of answers to questions about objects, their location, and the time of the last encounter. The system resolved ambiguity in descriptions for about 92% of objects, requiring only two additional images per object, and improved description accuracy by 5% compared with state-of-the-art methods.
"Unlike ChatGPT, for physical artificial intelligence it is not enough simply to warn that the robot may sometimes make mistakes. A robot's ability to know what it does not know will be necessary for the safe adoption of the technology," MIT's press release quotes senior author Luca Carlone as saying. In the future, the researchers plan to apply the method to other types of robots, including manipulators, and to extend uncertainty estimation to related tasks.