Humanoid robots, that is, machines with a body resembling a human one, could perform household work and other manual tasks. But to handle them, robots need to precisely coordinate movements, posture and object grasping. One way to teach them this is to show how a human performs a task and record their actions on video and sensors. However, collecting such data suitable for training algorithms is not easy.
A group of researchers from the Shanghai Artificial Intelligence Laboratory, Zhejiang University and Shanghai Jiao Tong University developed a wearable system for recording movements and created the EgoExplorer dataset on its basis. The paper was published as a preprint on arXiv, that is, before peer review.
The system includes a GoPro HERO13 camera on the head, five VIVE Ultimate trackers (devices that track position and orientation in space), MANUS Metagloves or Wuji sensor gloves, as well as 3D-printed mounts and straps. The camera films what is happening from a first-person perspective. The trackers are mounted on the head, the back of each hand and each ankle. The gloves record finger movement, and the Wuji model also records tactile sensations - data on contact and pressure.
Using this system, the authors recorded 100 hours of data while people performed various tasks, with an emphasis on professional activity. In the EgoExplorer dataset, first-person video is combined with a reconstructed three-dimensional model of whole-body movement, fine finger motor skills and tactile observations. Each step of a task is accompanied by a text description.
To reconstruct whole-body movement from sparse data from five trackers, the researchers created the Calibrate-Complete-Constrain software pipeline. First, the system is calibrated to a specific person and sensor placement, then it predicts the movements of joints that were not tracked directly, relying on learned patterns, and finally checks that the reconstructed movements are consistent with the sensor readings, remain smooth and match the body build.
Compared with other motion reconstruction methods, the new approach reduced the error in body point positions by 71.9% relative to the best of the compared variants. In preliminary training experiments, using video, motion and tactile data together gave better results than using them separately. The researchers also trained a controller for a simulated Unitree G1 humanoid robot to repeat two recorded human movements. The robot in simulation completed both sequences without falling, but this does not confirm precise interaction with objects or performance on unfamiliar tasks. The system was not tested on a real robot.
The authors note that in the future EgoExplorer could be useful for training other controllers in simulation, and then possibly for improving robot performance in real conditions. However, a finished product or mass application is still far away.