Audio Aid for the visually impaired using Amazon Alexa

Aspiration

We wanted to create something that would be able to help those who are visually impaired. We went with Amazon’s Alexa because it is a voice service. Although we have created our custom skill-set using the Amazon Echo, ideally, our add-on to Amazon’s voice service would be optimized for the Dot, as it is portable and better-suited for the aspects of our skill-set that help with navigation (and not as suited for a plugged-in speaker).

The first aspect of our skill-set involves navigation of a grocery store for finding desired items in aisles. The user can first specify which store they are at, and Alexa will then reference the user’s saved grocery items for foods that are available at the current store, and then give relative directions to the proper location in the store. Then, the second aspect of our skill-set comes into play.

The second aspect involves image recognition. We have a Nest camera that takes a picture when the user asks it to, and then passes the information to Microsoft Cognitive Services’ Computer Vision API to analyze it. It then returns keywords to describe the environment it sees. This aspect can help in identifying the correct items and determining item prices.

Together, Alexa and Nest can be used to fully navigate through an indoor location such as a grocery store or an academic hall.

Inspiration We were inspired by videos and blogs of blind people and the problems they vocalized. There were often mentions of certain grocery items (such as canned items) that were difficult to identify, as well as navigating unknown areas.

What it does It essentially provides the visually disabled with an indoor navigator and an image descriptor. This allows blind people to be more individually autonomous, whereas previously, many blind people would have asked a friend to help them learn a new area or identify items.

How I built it We used AWS to build Alexa’s functionality. Microsoft Cognitive Science’s Computer Vision API was used for image recognition. Alexa’s navigational component was built using JavaScript.

Challenges I ran into Working with Amazon’s vast collection of services was difficult. Making different components work with each other was a big challenge.

Accomplishments that I'm proud of We’re pretty proud of this as a whole. Getting the image recognition working was great, as was getting Alexa to properly respond to queries.

What I learned We learned about working with AWS and the vast array of options that it provides to developers. We also learned how to think from a different perspective to tackle problems that we ourselves do not encounter on a regular basis.

What's next for Audio Aid for the visually impaired using Amazon Alexa

We would like to build more to our functionality, such as perhaps video recording to identify traffic, and for more detailed voice descriptions of the surrounding environment.

Built With

• amazon-alexa
• amazon-dynamodb
• amazon-web-services
• lambda
• microsoft-cognitive-services
• nest
• node.js
• python2.7