Lip-reading is a difficult task, especially when it is noisy around. But researchers from the Korea Advanced Institute of Science and Technology (KAIST) have found a way to improve it with artificial intelligence. The new system first recognizes the words it is confident about, then uses them as a hint to decipher ambiguous parts of the phrase.
The technology could be useful where sound is unavailable or distorted: for example, to turn into text speech drowned out by noise, or to help people with hearing impairments follow a conversation.
A group led by Professor Yong Man Ro of the School of Electrical Engineering has developed a method that increases the accuracy of visual speech recognition (VSR) - the name for converting lip movements into text. It is based on a diffusion large language model (DLLM). It builds a sentence by gradually filling in gaps and can take into account words both before and after an unclear spot. The work is posted on the arXiv preprint server (a preprint is a scientific paper published before peer review).
The difficulty is that different sounds sometimes look almost identical on the lips. In English, for example, /p/, /b/ and /m/ are practically indistinguishable on video. Conventional models generate words strictly in order, and an error at the start of a sentence drags the following ones along. The Korean scientists changed the order: fragments with the clearest articulation are recognized first, and they then serve as context for the remaining positions.
Model training was split into two stages. First it learns to match lip movements in video with text, then to estimate the likely length of the sentence. During decoding, the model tries several length options, builds a candidate sentence for each, and chooses the most reliable one based on length likelihood and word confidence.
The LRS3 dataset was used for training - 433 hours of lip-reading video with precise transcriptions. The word error rate (the percentage of incorrectly recognized, missed or added words) was 19.4%. This is the best result among methods trained only on LRS3. The lower this figure, the more accurate the system.
In the future, the technology could be built into communication tools: for example, in noisy production it could tell what a worker is saying, or it could help the hard of hearing. The team plans to reduce the model's size and computational costs so it can run on wearable devices and robots without the internet. No timeline for a product release was mentioned.
"The study is valuable because it shows that flexible control of the order of word generation by itself overcomes a fundamental limitation of speech recognition related to visual ambiguity," a press release quotes Ro as saying. He also noted that the work will lay the foundation for physical AI that people can trust at home and in industry.
The first author is Dr. Jeong Hun Yeo of the same school, the co-authors are graduate students Chae Won Kim and Hyeongseop Rha, and Ro is the corresponding author.