People communicate not only with words. A raised hand, a change of posture or a facial expression help convey meaning and emotion. Now researchers from Tohoku University (Japan) and the University of Hong Kong have developed an artificial intelligence (AI) system that translates speech into coordinated body, hand and facial movements. This could make digital characters and virtual presenters more expressive and more like living people.
The system, called DynaConTalk, generates three-dimensional (3D) human movements that correspond to the spoken text. Unlike approaches that create only individual gestures, it coordinates different parts of the body and maintains smoothness of movement throughout long speeches. Users can also set a desired pose or provide examples of movements to guide the generation.
The results were published in the journal ACM Transactions on Graphics. The study is also planned to be presented at the SIGGRAPH Asia 2026 conference, which will be held in Kuala Lumpur (Malaysia) from December 1 to 4.
Creating realistic full-body movements from speech is not easy. Large changes in pose can obscure small movements such as finger gestures and facial expressions, and AI models have difficulty learning them. This leads to repetitive gestures or unnatural trembling. Another problem arises when joining separately generated fragments of movement: sometimes noticeable jumps in speed or rhythm appear.
To cope with this, DynaConTalk separates movement into slow and fast components, keeping them synchronized. This is similar to separating sound into low and high frequencies. By learning on different types of movement together, the system captures both large changes in pose and small brief gestures. In addition, it dynamically adjusts how much to rely on different characteristics of speech - rhythm, voice features and the spoken words - when generating movement.
The system also uses information about previously generated movements to guide the next fragment. This allows continuity to be maintained over long sequences, reducing abrupt transitions and making movement more natural during prolonged speeches.
The researchers evaluated DynaConTalk on two publicly available datasets of speech and human movement - BEAT2 and SHOW. They also conducted a study in which 30 participants compared videos generated by three different systems on 15 speech examples. DynaConTalk received the highest average scores in all four categories: naturalness, correspondence to the spoken text, synchronization and diversity of movements. Automatic evaluations additionally showed that its movements are closer to the distribution of recorded human movements.
The technology could find application in character animation for video games and films, virtual presenters, online education and remote communication. For example, creators could set a character's initial pose or provide a reference movement, and then let the AI generate the surrounding gestures and expressions. This provides a more flexible way to create animated characters whose movements complement what they say.
The team hopes to extend the approach to a wider range of full-body movements, including walking. However, existing datasets contain relatively few examples of large-scale movements, and speech alone is not enough to determine how a person should move through space. The researchers plan to study larger movement datasets and additional guidance on trajectories to overcome these limitations.