A new web application allows Alexa to have a conversation with the deaf

As practical as voice assistants are, these tools powered by A.I. have fallen behind in their inability to serve the deaf community. But an enterprising computer scientist has come up with a solution that will allow those with hearing problems to experience the ups and downs of Amazon Alexa, Google Assistant and Siri with all the others. Abhishek Singh, who grew to prominence when he built Super Mario Bros in Augmented Reality, created a web application that uses a camera to read sign language and then translate those words into spoken language for an Amazon Echo. When the echo speaks of its response, the whole process unfolds backwards, giving a typed answer for the hard of hearing or the deaf to read.

This application goes beyond the pseudo-solution provided by the Amazon Echo Show. screen to the smart home hub to allow members of the deaf community to interact with the smart home assistant. However, this still did not allow Alexa and her user to really have a conversation – Singh's offer, however, corrects that.

"The project was a thought experiment inspired by the observation of a trend among companies to create instant and transparent interactions," he told Fast Company . "If these devices are to become a central way of interacting with our homes or performing tasks, then a reflection must be made to those who can not hear or speak. The seamless design must be inclusive in nature. "

In the construction of the system, Singh formed a AI system in the Tensorflow machine learning platform, signing words through his webcam ad nauseam to" teach "the language He added Google's speech synthesis capabilities to translate sign language into spoken words.

Although Singh's solution is elegant, the coder still hopes that Amazon will finally recognize sign language alone. "It's where I hope it's going. And if this project leads to a small push in this direction, then the mission accomplished, "he said." In an ideal world, I would have built it directly on the show, but the devices are not still hacked, [I] was not able to find a way to do it. "