Robots make mistakes more often than humans, even when performing simple household or factory tasks. One reason is that they have difficulty quickly extracting the needed details from past experience and acting in real time. Even so-called vision-language models (VLMs), which understand images, video and text, do not always help.
Recent approaches used AI agents - semi-autonomous programs based on VLMs that decide when a robot should analyze specific frames from a long history. But this led to errors, high computational costs and slow reactions.
Researchers from the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) and the Machine Learning Department at Carnegie Mellon University proposed the Workspace Models approach. Instead of searching for data on the fly, large VLMs perform reasoning in advance, offline. During this process they train a compact module - the Workspace Model itself - which enables a robotic arm, for example, to plan its next movement in milliseconds in response to changes in the situation. The work was published on the arXiv preprint server.
The Workspace Model learns to remember key details by transforming the history of observations into multidimensional data points - "latent vectors" that encode information useful for predicting future actions. Two VLMs are involved in training: one reviews video, the other extracts spatial data from images. The Workspace Model compresses observations into latent vectors with this "meaningful" information, and then trains another small model - a policy that predicts actions from these vectors.
At test time, the Workspace Model and the policy work together, providing a fast reaction to what the robot's camera sees. The VLMs have, as it were, passed "muscle memory" to a lighter model. The robot can quickly put a block back in a box simply by reacting to changes in visual and positional inputs.
Lead author Nitish Dashora of CSAIL notes that the system provides a significant improvement in latency - the robot's reaction time - compared with agent-based approaches. In experiments, virtual and real, on a robotic arm, the system successfully picked up and put away blocks in about 92% of cases - about 25 percentage points higher than the next best approach, which used an agent to search for key frames. Among models capable of reasoning with long-term memory, the Workspace Models system proved to be the fastest.
According to the researchers, this approach could be useful in a dynamic environment, for example in a kitchen, where a robot needs to quickly recall important details when performing tasks that require memory. The scientists also suggest that AI agents could act as "meta-learners", choosing which "muscle memory" skills to pass to the robot, instead of directly controlling it.
Going forward, the team plans to improve the system's efficiency and accuracy: it currently processes long sequences of input data, which increases computation, but a recurrent approach could reduce the load. In addition, to improve spatial understanding and task performance, the researchers intend to train the Workspace Model on VLMs specialized for robotics.