Brain signals translated into speech using artificial intelligence

In an effort to provide a voice for people who can not speak, neuroscientists have devised a device that can turn brain signals into words.

This technology is not yet precise enough for use outside the laboratory, although it can synthesize whole sentences for most intelligible. Its creators have described their speech decoding device in a study¹ published on April 24 in Nature.

Scientists previously used artificial intelligence to translate simple words^2,3Chethan Pandarinath, a neuroengineering engineer at Emory University in Atlanta, Georgia, co-authored a commentary accompanying the study. "Making the jump from one syllable to one sentence is technically quite difficult and is among the things that make the current work so impressive," he says.

Mapping of movements

Many people who have lost the ability to speak speak using technology that forces them to make small movements to control a cursor that selects letters or words on a screen. The British physicist Stephen Hawking, suffering from motor neuron disease, is a famous example. He used a speech-activated device powered by a cheek muscle, said study leader Edward Chang, a neurosurgeon at the University of California at San Francisco.

As people who use such devices have to type the words letter by letter, these devices can be very slow and produce up to ten words per minute, Chang explains. The natural spoken speech is on average 150 words per minute. "It's the effectiveness of the vocal tract that allows us to do that," he says. Chang and his team decided to model the vocal system when building their decoder.

The researchers worked with five people who had electrodes implanted on the surface of the brain as part of the treatment for epilepsy. First, the team recorded brain activity as participants read hundreds of sentences out loud. Then, Chang and his colleagues combined these recordings with data from previous experiences that determined how the movements of the tongue, lips, jaw and larynx created sound.

The team formed an in-depth learning algorithm on this data and then integrated the program into their decoder. The device transforms cerebral signals into estimated movements of the vocal tract and transforms these movements into synthetic speech. People who listened to 101 synthesized sentences could average 70% of the words, Chang explains.

In another experiment, the researchers asked a participant to read sentences out loud and then to mimic the same sentences by moving their mouths without producing sound. The sentences synthesized in this test were of lower quality than those created from audible speeches, explains Chang, but the results are always encouraging.

Future intelligible

Stephanie Riès, a neuroscientist at San Diego State University in California, more easily understands the word created by associating brain activity with movements of the vocal tract and translating them into sounds.

But we do not know if the new speech decoder would work with words that people only think, says Amy Orsborn, a neural engineer at the University of Washington in Seattle. "The paper does a very good job of showing that it works for an imitated speech," she says. "But how would it work if someone does not move their mouth?"

Marc Slutzky, a neurologist at Northwestern University in Chicago, Illinois, shares this view and says the decoder's performance leaves room for improvement. He notes that listeners identified the synthesized speech by selecting words from a set of choices; as the number of choices increased, people had more and more trouble understanding words.

The study "is a very important step, but there is still a long way to go before the synthesized speech is easily understood," says Slutzky.