selwyn

• 

  Our hardware

Inspiration

We have a friend who suffers from hearing loss which puts him at a disadvantage while playing competitive video games.

What it does

It is a project designed with accessibility at its core. It seamlessly integrates into the gaming experience for those who are hard of hearing or deaf by removing barriers to enjoyment and reducing frustration.

How we built it

Server

A web server runs on the user’s computer that collects audio and streams it to the client. Audio is collected by configuring a new source to loopback the default audio sink into the new source and reading wave frames from it as bytes and sent out using UDP

Client

The client is a Raspberry Pi with an LED matrix that initiates a connection with the web server and collects audio, converting it into two volume measurements corresponding with the left and right channels in stereo audio and animating on the LEDs.

We read the stream back from the server for the LED matrix, we then map the volume representation to the matrix elements. We drive the LED matrix using I2C protocol on one bus. The logic for writing bytes to the LED matrix address in order to light it up is dictated by volume thresholds in our program.

Challenges we ran into (skippable)

Creating a loopback audio source

This was our first time working with audio in code, so it was difficult figuring out the specifications of the audio system. There’s no way of directly recording from an audio sink so a loop back must be created. We solved this by using Pulseaudio Volume Control and replacing the audio source for the server with the global audio sink.

Raspberry Pi over Ethernet

We had a lot of trouble sshing into our Pi over Ethernet, and we didn’t have a mouse or keyboard for a while, so it took a long time to figure out what the problem was exactly. The solution was to set a static ip address for both the Pi and the laptop.

Separating Audio Channels

This took some research into how wave files worked, and we found out that wave files with multiple sound channels are interleaved, so for a 2 channel file, the left channel is stored in even indices and the right channel is stored in odd indices.

Sending Audio Data over UDP

Our client and server refused to connect using UDP, so to help debug I spun up a quick TCP server using flask and using Wireshark, realised that requests to the server were responded to with RST. Trying totally wrong port numbers yielded the same result, so I instead looked at whether the port was actually being exposed. It turns out I need to bind the server to the address as it appears to the pi rather than just localhost. It turns out this wasn’t the problem for the UDP side anyways, and that the UDP sockets were misconfigured.

Moving from Beagle to Pi

We had to do some rewiring due to the differing pinouts but the Adafruit backpack on the HT16K33 chip is relatively straightforward to work with when rewiring, since it only needs four pins in total for the power and GPIO.

Enabling I2C on certain GPIO

How to represent volume numerically and physically

How we represent volume numerically was very important. Different scalings yielded different sensitivities and gave either more or less information. We tried linear, logarithmic, and exponential scaling, hypothesizing that linear and exponential scaling might improve sensitivity at lower volumes, but logarithmic with a running average to smoothen values worked the best.

Mapping/Addressing for the matrix and writing bytes efficiently

Bytes can technically be written to the matrix address without using i2cset, possibly instead as a write command that writes multiple bytes (up to 16 for an 8x8 matrix's rows). We did not implement this due to the time restriction but this is a possible improvement.

Accomplishments that we're proud of

We managed to get a working MVP! There were a good amount of moving pieces and we managed to bring it all together to meet our requirements.

What we learned

• Interfacing with I2C protocol
• Networking
• PyAudio
• Reading/writing WAV files
• Subprocess

What's next for selwyn

We would like to apply our project to a larger scale with larger LED matrices, and have 2 of them so that the display is more intuitive to how one would visualize directional audio. Audio is often hard to classify, and our project only represents volume, so we can integrate audio classification to provide more information about the source of a sound in addition to its proximity from the user.

Built With

• adafruit
• alsa
• beaglebone
• c
• python
• raspberry-pi
• shell
• udp