The SciFi lab team at Cornell University, led by associate professor Cheng Zhang, has developed the EchoMotion system, which determines a person's body position from reflected ultrasound. It uses the built-in speaker and microphone of ordinary smartwatches. The work was presented at the UbiComp/ISWC conference in Shanghai and published in the Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.
Unlike existing wearable devices, which track only the movement of the gadget itself using accelerometers and gyroscopes, EchoMotion captures the movement of the whole body. The system sends out inaudible ultrasound signals and picks up their reflections from the arms and legs, and then a trained neural network converts these echo signals into a three-dimensional pose. This approach preserves privacy - no cameras are needed - and requires less data for processing than video tracking.
First, the researchers collected data from 10 volunteers who performed 18 exercises with the watch on their non-dominant hand. The prototype was 3D-printed and had a pair of microphones and speakers on each side. Accuracy was checked by comparing the predicted positions of 12 joints (from the head to the ankles) with poses captured by cameras.
The system was then tested on production Samsung Galaxy Watch 6 devices with 15 new participants. The average error in determining the position of each joint was less than 10 centimeters. A single 12-minute calibration reduced the error by more than a third - to about 6.1 centimeters. The same model, without retraining, worked on the Galaxy Watch 7 and Google Pixel Watch 3.
Outside the lab, 10 new volunteers performed the same exercises in a small room, in a narrow corridor and next to a noisy road. Furniture and extraneous noise did not interfere, but in the corridor accuracy dropped slightly because of echoes from other people. Co-author Saif Mahmud suggests that with more data - from thousands of users instead of dozens - the model will learn to cope better with crowded places, such as a subway.
In the prototype, the sensors consumed about 12 milliwatts - roughly 40 times less than a wearable device's camera. A 30-minute session on the Galaxy Watch 6 used 5-6% of the charge. The team is now developing the technology for fitness tracking: a virtual trainer will be able to count repetitions and correct exercise technique. The combination of EchoMotion with cameras on smart glasses for tracking the objects a person interacts with is also being studied.