UBC researchers to explore tech that generates speech from forearm muscle signals

A team of UBC researchers has been awarded $150,000 USD from Meta Reality Labs to develop a muscle-based voice control interface. The Brain2Speech research group, led by Electrical and Computer Engineering Professor Dr. Sidney Fels, is one of six global teams selected to receive funding to develop optimal and responsible strategies for exploring neuromotor signals for human-computer interaction.

The proposed project, sEMG-Talk, is an interface that uses muscle signals, or surface electromyography (sEMG), and machine learning to generate speech in real time. A device on the user’s forearm contains high-density sensors that control a virtual vocal tract model, allowing the user to speak or sing without the use of the mouth and throat. The technology builds on Dr. Fels’ influential Glove-Talk II, an innovative system that uses neural networks to map hand gestures to speech. 

“sEMG‑Talk lets users learn to ‘play’ their voice like an instrument, and with ethics built in from day one, we can ensure the technology truly serves those who need it,” said Dr. Fels.

The project presents a unique opportunity to engage with the important ethical implications that come with high-bandwidth human-computer interactions. With the guidance of Dr. Julie M. Robillard, associate professor of neurology with the UBC Faculty of Medicine, the team seeks to build a taxonomy of the neuroethical considerations emerging as sEMG technology progresses. This taxonomy will inform device design from the earliest stages.

The team will spend the next 18 months creating the proof-of-concept sEMG-Talk system and conducting user studies where participants will learn to speak and sing using the interface. While still in the early stages of development, the interface has potential applications in assistive technology, gaming and digital music creation. 

Much of the Brain2Speech research group connected through the UBC Language Sciences Institute, a network of researchers in language-related fields. In addition to Dr. Fels and Dr. Robillard, the interdisciplinary team includes:

• Dr. Lara Boyd, UBC Department of Physical Therapy
• Dr. Bryan Gick, UBC Department of Linguistics
• Dr. Jing Xu, University of Georgia Department of Kinesiology