Therapute

Inspiration

In today’s world, accessing comprehensive physical and sports therapy is challenging due to high costs and limited support between sessions. Traditional methods often involve spaced-out appointments, leaving patients reliant on at-home therapy exercises to advance their rehabilitation. These exercises may include potentially risky maneuvers such as dumbbell thrusts, deadlifts, or weighted internal rotations, which, if performed incorrectly, can impede muscle regrowth, hinder recovery, and cause reinjury. This lack of personalized guidance not only prolongs the rehabilitation process but also poses risks to patients' physical and mental well-being.

What it does

Our solution, Therapute, aligns with the United Nations' Sustainable Development Goal of ensuring healthy lives and promoting well-being for all, at all ages, specifically targeting the decrease in the proportion of the population with musculoskeletal disorders, one of the most common disorders in the worldwide (especially in the Western world). We were motivated to address this goal and target due to the exorbitant costs associated with physical therapy, the high rates of reinjury during PT, and the lack of cohesive platforms to bridge at-home exercises to spaced-out physical therapy sessions. By tackling these challenges, we aim to contribute to global well-being by promoting healthier, less pained individuals worldwide, fostering innovation and productivity as more people can actively engage in creating solutions to global problems. With our custom time-warping algorithm, Therapute is able to analyze the exercise form of patients and provide them with valuable intuitively visualized insights describing their errors and showing them an improved path to recovery. Patients are able to document any pain or difficulties they face during their exercise with our LLM model. The patient's physiotherapist is able to track their patient's progress and keep track of important insights such as their most common exercise errors and effectively analyze their trajectory on their path to recovery and diagnose/prevent potential reinjuries early on.

How we built it

To build and host our application, we utilized a range of technical components in our architecture. The frontend was crafted with Next.js, providing a user-friendly, interactive interface. Upon login, user data, including common mistakes and assigned exercises, is fetched from Firebase and displayed in real-time on the frontend, enriching the therapy process insights. Videos uploaded by users for exercise reps are sent to our Flask backend server. Here, our machine learning framework is deployed, beginning with frame-by-frame video loading via OpenCV. MediaPipe API detects keypoints within the user's body, facilitating our pose analysis logic for error identification. We categorize mistakes into static (applying to the entire rep) and dynamic (occurring at specific time frames), analyzing them using geometric vector analysis and dynamic time warping. Identified mistakes trigger modification of video frames to highlight key points, alongside error/feedback messages. While our time warping algorithm currently only works for dumbbell thrust, expansion to other exercises is underway and we have developed models already for static mistakes of other exercises. Annotated videos are stored in Google Artifact Registry, enabling users to reference past sessions' mistakes/improvements. For accessibility, we've containerized the app with Docker for local deployment and are progressing towards Google Cloud Run integration. Firebase - Store user specific data such as most common mistakes, assigned exercises, past annotated sessions video files

MediaPipe - Identify keypoint coordinates of joints on a user’s body frame by frame as they are performing their specified exercise

Cloud Run (Still being implemented/debugged) - Utilizing for hosting our application and ensuring our solution is accessible to anyone without having to run it locally and deal with issues with limited computational resources (Still working on it–the frontend is hosted while the backend is having a few bugs with user video upload)

Artifact Registry - Store annotated videos as processed by our machine learning pipeline for users and physical therapists to reference during future sessions Challenges we ran into While scaling our application, we realized that we needed a new solution for efficiently storing video files recorded by patients. In order to maintain an organized file base and maintain cybersecurity protocols to protect the security of patients who upload videos to Therapute, we decided to implement a cloud solution. We faced issues with integration initially as file retrieval was too extensive of a process and lengthened the analysis period significantly. We transitioned to utilize Google’s Artifact Registry and found the service to work efficiently in our architecture. Its ease of integration with other Google products such as Cloud Run (which we are still implementing for hosting) made the service a perfect fit for Therapute.

Accomplishments that we're proud of

We're proud of creating an app that can transform the lives of patients seeking physiotherapy. Our inspiration for Therapute stems from firsthand experiences witnessing the debilitating effects of musculoskeletal conditions on family members and recognizing the critical importance of swift and effective intervention for both individual well-being and global development. After consulting numerous physiotherapists at 6 clinics across the Atlanta and Orlando Metro areas, we were able to pinpoint our main action items as creating an effective model for detecting form errors and equally importantly, designing a system that makes such errors intuitive for patients to understand and learn from. We are proud of being able to deliver on this MVP and expand Therapute to transform the rehabilitation process; we are getting closer to our goal of enabling physiotherapists to gain deeper insights into the quality of their patients' exercise routines and better track their progress over time, decreasing the risk of reinjury.

What we learned

We learned an immense amount about hosting and model deployment as this was our first time creating a full-stack web application that runs a model on the cloud. Apart from the various Google Cloud products that we learned to use through this project, we also learned about the physiotherapy industry and the kinds of insights that would be useful for patients and therapists alike. From our talks with med tech experts at Elevance Health, we were suggested to improve our model by utilizing dynamic analysis rather than static analysis which we utilized when initially creating Therapute. We began to focus on creating a model that analyzed a patient's range of motion rather than checking for static instances when set thresholds were broken. This allowed us to provide a level of insight that is similar to the methodology through which a physiotherapist would assess their patients. This understanding allowed us to increase the value of our product multifold as Therapute was now able to simulate the level of judgment that a physiotherapist would utilize, allowing us to get closer to our mission of providing the best exercise form correction and injury mitigation for patients right from the comfort of their home.

What's next for Therapute

Moving forward, the next steps for our project involve collaboration with physical therapists to introduce our app into beta testing trials. By partnering with professionals in the field, we can gather real user data and utilize it to enhance our models, ensuring accuracy and efficiency in assessing exercise performance. Additionally, expanding the repertoire of exercises covered within the app is crucial. This will not only improve the platform's utility but also attract a larger audience seeking diverse exercise options. To reach a broader audience, we plan to further expand our solution in several ways. Firstly, incorporating a wider range of exercises onto the platform will cater to the diverse needs of users, making it a comprehensive tool for various fitness goals and rehabilitation requirements. Partnering with multiple physical therapy clinics will enable us to assess their clients' satisfaction with our product, building trust and credibility within the healthcare community. Moreover, establishing a strong customer base through targeted marketing and outreach efforts will be essential before launching the platform into the market. By strategically expanding our solution and fostering partnerships, we aim to reach a larger audience and make a positive impact on individuals' fitness and rehabilitation journeys.