-
-
logo!
Inspiration
We found a terrifying statistic. An American study found that, for every minute a person is left on hold by emergency service operators, 1000 people will die.
In contrast to the fact that, during COVID, in Mississauga and Brampton, the average wait time peaked at 6 minutes. This is terrible, and can cost many, many lives.
Though there are many root causes that cannot be directly addressed by technology, such as an influx in population, or an increase in unnecessary/non-emergency related phone calls.
However, The Mayday Project, aims to fix some of these concerns.
What it does
The Mayday Project is an AI-based phone calling service. Should all operators be unavailable, the AI will ask if the user is able to speak to an AI while waiting for the operator. If they confirm, they will be prompted to describe the situation to the AI, then wait for the operator. This information is passed through a very specific pipeline, at the end, extracting the location, time, and summarizing the information from the caller, displaying it directly on a dashboard for the operator. Thus, the operator will be able to better and much faster assist the caller, with a good majority the information at their hands.
How we built it
The project is run via a Flask server interacting with a Twillio API/phone number. Sadly, due to both legal restrictions (impersonating police/emergency services) and monetary restrictions, the phone number is not open for demo. However, a demo is provided by video, where it clearly works in real time. The caller is presented with the message asking them if they'd like to speak to an AI. If they confirm, the AI prompts them to describe the situation with as much detail as possible. Once the caller is done describing the situation to the best of their ability, they press the # key, and the Twillio API returns a wav file. This is run through Google's Cloud Speech-to-Text. After the text is recieved, it is processed via a HuggingFace model (BART and RobeRTA) to extract the location, and summarize the call and important signifiers. Finally, all of this information is passed through Google's BERT to extract the similarities between certain incidents. This can heavily reduce a big factor of why operators can get bogged down by calls - multiple calls about the same incident. If the BERT model detects a similarity greater than a certain threshold, it assigns a random "Group" string identifier to all of the incidents detected, grouping them. All of this is saved to Google's Firebase, from where a React app (which is live! link can be interacted with, after logging in via OAuth. The server is populated with fake data.
Challenges we ran into
Mostly we realized the efficacy of being able to fine tune models. Without being able to fine-tune models in such a short time frame, we realized how important it is. However, we found ways to circumvent the issues to have reasonable standard for a MVP.
Accomplishments that we're proud of
Learning Twillio API and having a callable phone number that can easily and quickly process voice calls + hosting a Flask server via ngrok, and making sure it can read and write to FireBase, and then linking it all to a pretty FrontEnd was a ridiculous achievement for us. Having accomplished a fully functioning prototype, and a project website link + a live demo was really fulfilling.
What we learned
We realized the importance of communication between teammates working on different parts of the project, and the importance of reading documentation thoroughly before developing. Although it may seem intuitive, it's easy to forget, and can cost heavily.
What's next for Mayday - an AI based emergency system operator
Improving the efficacy of the AI models, fine-tuning BERT and BART would be incredibly important. Apart from that, a deeper end-to-end understanding of transcription and location extraction. A feature we didn't have enough time to implement was to implement the Google Maps SDK to actively visualize hotspots for calls based on these transcriptions and incident reports.
Log in or sign up for Devpost to join the conversation.