Kinexote

Inspiration

Our team wanted to create an interactive art installation that connected physical body movements to musical sounds and on-screen visuals. To do this, we drew inspiration from digital music production and asked what a gesture tracking camera could do within that realm. The result became Kinexote!

What It Does

Kinexote tracks arm movements and gestures to select and customize the audio you input to your liking—for example, applying reverb, filters, or different amounts of gain. The audio output then affects visuals on screen, from various amplitude representations to different colors and shapes based on frequency content.

How We Built It

An Xbox Kinect was interfaced with the open source Processing programming platform to track the user's arm and hand position to alter the audio. We also tracked certain specific gestures, such as closing and opening the fist to start/stop recording audio. After the tracking data is received, the Minim library contained within Processing was used for Digital Signal Processing, e.g. manipulating sound clips that are inputted into the program. We used a Shure SM57 and Focusrite Scarlett 2i2 for the hardware to input audio.

Challenges We Ran Into

Because this was quite an abstract and novel idea, the team ran into communication difficulties early on in the process in describing to each other what we wanted. However, this could be attributed to an excess of creativity within our team, which when paired down and structured, lead to great results. Some technical challenges involved a lack of experience with Processing and with the Minim library (which lacked clear and robust documentation), and integrating all of the subsystems (audio, kinect-interfacing, and UI) proved to be difficult as well.

Accomplishments That We’re Proud Of

We are proud of creating a multimedia project that enables user to explore their own creativity in sound, visuals and art! It was awesome seeing people’s reactions towards the project and how much fun everyone had trying it out.

What's next for Kinexote

We are hoping to brush up the technical aspects of the project such as interfacing between different libraries and revising the user interaction process. We feel that there are other intuitive methods that we can incorporate to make the experience more user friendly. We also want to implement more functionality, whether that be audio-related or visual/UI related.

Built With

• dsp
• java
• kinect
• kinectpv2
• minim
• processing