Inspiration
The ability to measure breathing activity from within VR can offer a multitude of practical applications, particularly in a healthcare setting. Breathing data can give insights into a patient’s psychological and physiological state, while VR can be used to influence the patient's mental state or evoke certain behaviors from the patient. By combining respiratory monitoring and VR, a VR experience could be dynamically adapted based on real-time breathing data, patients could be encouraged to breathe and behave in a certain way, and breathing and other data could be collected for later analysis. Despite these potential benefits, there are challenges in combing VR and respiratory monitoring. Current methods of measuring breathing in VR either use non-contact sensors that provide little valuable information, or require invasive sensors that are uncomfortable for the user. This led us to create our own method of respiratory sensing for VR using thermal imaging.
What it does
This project presents a non-contact respiratory sensor for VR. This sensor combines a small thermal sensor and a thin medium; the thermal camera monitors the heat signature on the thin medium as the user exhales onto it, and the thermal signatures are processed to extract real-time respiratory information. The sensor is mounted on the front of a VR headset so that the sensor is in the path of the user's exhaled air, but the sensor does not physically touch the user. Though the sensor is non-contact, the thermal signature on the medium can provide more data about how the user is breathing than traditional non-contact sensors; metrics such as breathing rate and pattern can be extracted from the thermal signatures, as well as other information such as whether the user is breathing through their nose or mouth. To demonstrate the possibilities, we've created VR sample applications using this sensor that demonstrate how a user can use their breath as a way of interacting with the virtual environment.
How we built it
Hardware
We use a Meta Quest 2 headset, a small Seek Thermal camera as our thermal sensor, and paper as our medium for our proof-of-concept. The thermal camera connects to the Quest 2 headset via USB-C. Our prototype attaches the camera and medium to the headset via LEGOs, but we plan to 3D print a smaller attachment and use a different medium type in future iterations.
Software
The software was built using Unity and C#. Seek Thermal provides an Android SDK that we used to create an Android plugin for Unity. We use this plugin to build applications for Meta Quest devices that use Seek Thermal cameras as input.
Challenges we ran into
The Seek Thermal camera we used has a narrow field-of-view (FOV), which isn't ideal for our setup where the thermal sensor is viewing a medium from a very short distance. Additionally, the camera's low framerate (9 Hz) isn't ideal, nor it's small image size. However, Seek Thermal makes other cameras that have a wider FOV, higher frame rate and larger image size. We would like to replace our current camera with one more suited to our project.
Accomplishments that we're proud of
We're proud of the fact that we've created a working prototype and demo applications that show that we can monitor and utilize breathing data from within VR. We're also proud of the fact that we created a working Unity plugin from Seek Thermal's Android SDK, and can use this to create a variety of breathing-based VR applications.
What's next for Breathing Monitoring in VR using Thermal Imaging
We plan to create a new sensor attachment that is smaller and cleanable. We also plan to investigate other medium types that can be easily put on and removed from the apparatus so that we can easily change the medium between patients.
Log in or sign up for Devpost to join the conversation.