Master Marimba

Inspiration

As a result of the COVID Pandemic, many music students could not attend school to continue learning about Marimba because they had to be sheltered in their homes due to safety and hygiene measures to combat the virus. And this caused them to miss classes since the students must practice with the instrument physically. It is for this reason that to prevent them from missing many classes, I decided to inspire the project to develop a Mixed Reality application that uses the capabilities of the technology offered by the Quest Pro to motivate students to continue learning and perfecting their techniques.

What it does

The application allows you to interact with a scale marimba using your hands to take the mallets. When making contact with each piece of the marimba allows the real sound to be emitted as if it were interacting with the instrument in a real way.

How we built it

23 musical notes were recorded in mp3 format. Each note represents a unique sound that is emitted when there is a collision between the mallet and the key found on the marimba. In our project, we worked with a real Marimba model of those currently used in schools around the world.

The project was developed in Unity using version 2022.3.16f. The SDK was initially used in version 62 and then we ended up using version 64, which is the latest available.

Then the capabilities of:

Hand Tracking.

Controller Tracking.

Grabbable objects and some scripts.

Using the Controls to take the mallets from a distance

Using controls for movement on stage

Passthrough Layer to view the external environment

Spacial Audio to provide a better user experience.

Box Collider and Rigbody on each key of the Marimba

Script to control the collision and detection of objects.

OVRCameraRigInteraction for visualization of the entire scene

FBX model and some textures to simulate the instrument.

Movement SDK to allow the user to move around the scene using teleportation either by activating the thumbstick

Challenges we ran into

Adjust each decibel per recorded musical note

Adjust user scrolling when using the SDK

Incorporate scene recognition.

Develop a script to control the speed between the hands and track them so that the person has the sensation that they really have a mallet in each hand and that it can be used to emit the sound.

We carried out several tests in a marimba school to find out the opinions of the teachers and students. In such a way that they will provide us with some type of feedback to improve the model and know the potential and acceptance of the project.

Accomplishments that we're proud of

Conduct tests with students who have different levels of learning regarding the marimba instrument.

Demonstrate that with the capabilities available for Meta glasses, it can be seen that it is possible to develop a mixed reality application that can be used to learn to play the marimba as an instrument and at the same time have fun throughout the teaching process. In such a way that the student has fun while learning, changing the traditional teaching paradigm a little.

What we learned

The marimba is an instrument that has its origins in Africa. Currently, it is played and practiced in countries such as the United States, Guatemala, Mexico, Colombia, Nicaragua, Costa Rica, and Ecuador. And in all these countries many music schools teach marimba.

There are two types of marimba: the simple marimba, which has a single keyboard (diatonic), and the double keyboard marimba (chromatic). Any type of music can be played on the double keyboard marimba.

Work with Meta XR all-in-one SDK capabilities.

Configure Unity to develop a Mixed Reality application

Generate the apk and test it on the Quest Pro using the Oculus Link Cable and then without the cable once the apk is installed using the Meta Quest Developer Hub

What's next for Master Marimba

Add a graphical user interface that provides three levels of learning: basic, intermediate, and advanced. For each level, the user will learn how to learn to play the marimba following a step-by-step guide to learn the organization of the sound, combinations, and tones.

Add the double keyboard marimba so that users can play any music of any rhythm using the Marimba. In case the 3D model is added, we would have to add the new notes.

Meet with a sound engineer or percussionist to help us improve the audio quality of musical notes. Adjust the pitch to its most appropriate level and eliminate any noise that affects the sound quality.

Allow the user to personalize their marimba through the use of Skins.

In the future, I would add the option of allowing 4 users to enter a shared room where each one will have an assigned marimba and can learn together in real time.