The EMO robot is learning to speak and express emotions.
Researchers have developed a robot named EMO that learns to speak and synchronize its lip movements with speech by observing itself and analyzing videos of people. While the technology is still imperfect, the robot becomes increasingly natural in communication as it gains experience.
Ingenium
When it comes to robots that closely resemble humans, one of the key features is the ability of their lips to move in perfect sync with spoken words. The new robot not only possesses this capability but can also independently learn to speak like a human.
How the EMO Robot Works
The EMO robot was developed by graduate student Yuhan Hu, Professor Hod Lipson, and their colleagues at Columbia University. The device consists of a head equipped with 26 miniature motors located beneath a flexible silicone facial skin. By using various combinations of these motors, the robot’s face can display a wide range of expressions, and its lips can form different shapes.
The Learning Process
The researchers began by placing EMO in front of a mirror. The robot randomly generated thousands of different facial expressions while observing itself. This allowed it to understand which motor combinations produced specific visual movements. This approach is known as a “vision-to-action” language model (VLA).
Next, EMO watched countless hours of YouTube videos featuring people talking and singing. This helped the robot establish a connection between mouth movements and particular speech sounds. The artificial intelligence system combined this knowledge with what it had learned through the VLA model, enabling EMO to synchronize its lip movements with spoken words using a synthetic voice module.
Current Capabilities and Future Prospects
At present, the technology is far from perfect: EMO still struggles with certain sounds, such as “B” and “V.” However, as it continues to practice and gain experience, its ability to hold natural conversations with people will improve.
Yuhan Hu notes: “When the ability to synchronize lip movements is combined with conversational AI, like ChatGPT or Gemini, it brings a whole new depth to the connection between robots and humans. The more the robot observes human conversations, the better it will mimic subtle facial gestures that allow us to connect emotionally. The longer the conversation context, the more context-aware these gestures will become.”
Scientific Publication
The results of this research were recently published in the journal Science Robotics.