CrimiDect

Helps to detect criminal conversations in less than 2-3 seconds. CrimiDect can transcribe audio and text conversations, scan through them, and tell if there’s something suspicious being exchanged.

Comment

Inspiration

We were inspired by the detectives around the world and their effort. We have all watched crime shows, and we noticed how long it takes for detectives to review transcripts/audio of conversations. After learning about the theme of the project, we quickly came to a consensus that detectives have very difficult jobs. For this reason, we wanted to build a software that utilized modern technologies, such as machine learning and artificial intelligence, to aid the often grueling tasks detectives have to complete. More specifically, we have all watched crime shows, and we noticed how long it takes for detectives to review audio and transcripts of conversations. We also realized that bias can also play a large part in the analysis of these conversation messages. To combat this issue, we wanted to make something that could analyze conversations without bias, and do it in an instant. This immediately led us to thinking about a machine learning model that could help these detectives. Our plan to train such a model was to utilize two different types of messages that detectives have to sift through: good and fraudulent messages. This is why we were motivated to use a supervised learning algorithm to train a machine learning model with two labeled datasets, one with fraudulent messages and the other with good messages, that would be able to classify new phrases detectives send to our software.

What it does

CrimiDect is a website that we created that detects whether or not a conversation has malicious intent. The website states our goals, impacts, and users and has an embedded google form. In the google form, users can submit an audio file and a transcript of their conversation along with an email. This will be sent to our machine learning algorithm, where we set up a neural network that can analyze the conversation. After the conversation is analyzed, a report is sent to the submitted email address, and it shows whether it detected the conversation to be criminal or good. Only sentences that are detected to be malicious are alerted to the user’s inbox, making it easier to go through every instance or criminal activity. From the submission of the form to the report being sent, this process takes 5 seconds, which makes it not only accurate, but also efficient.

How we built it

We used a plethora of technologies to create our product. For our backend, we used python with libraries like tensorflow and sklearn to create a deep learning model. This model was trained with over 1000 phrases that were both criminal and not. After repeatedly training the model with about 200 epochs, or training iterations, we created a form and backend to our project with an accuracy of 97.9 percent. The inputted data involved speech recognition using the speechrecognition API and libraries like pydub. We also used the Google APIs a lot including Google Sheets API and Gmail API by setting it up on our google cloud platform (dashboard). This made it possible to send the results to the user’s inboxes directly, seamlessly, and within a few seconds. There were also JSON files used to save and operate on data, which was good for our API keys. In our frontend, we made use of ReactJS and TypeScript to create a great UI/UX website that hosts the form. The project used modules like framer motion, NextJS, and ThreeJS. For decoration, we used Tailwind CSS.

Challenges we ran into

Our project was very successful in the end, but we ran into many challenges during the development stages. Firstly, we had many issues during the time we spent trying to create and train the machine learning model. For example, an issue we quickly realized was that, after trying to train our model a few times, our model was very inaccurate with analysis of future messages. In order to combat this, we had to analyze trends during the training stage of our model by creating graphs of the accuracy rate of the epochs as the training went on. The patterns in these graphs led us to realize that there was still room for growth in the model's accuracy of classifying new messages, so we decided to increase the number of epochs being tested. This change led to our model’s accuracy skyrocketing to over 97%, and it helped resolve the issue we were facing and gave us accuracy with expressions and tone detection. We also realized that some folders were different between devices, like node_modules. Additionally, client secrets could not be pushed to GitHub. We then found out about a .gitignore file that solved our syncing conflicts by ignoring certain files and folders.

Accomplishments that we're proud of

We are extremely proud of creating and training a deep learning model of such high accuracy and efficiency in such a short time period. This was one of the first things we did, and seeing it accurately classify phrases like “I will kill you” motivated us for the rest of the project. We are also proud of the speech-text. Since the API didn’t work out of the box and needed configuration, seeing our words appear on the screen just from a google drive file was borderline magical. Lastly, we are also pleased with how smooth the Gmail API went with sending the result’s email through the google cloud dashboard. We encountered a lot of problems with this relating to privacy errors and various restrictions, but at the end of the day, we got it to send emails almost instantly. Overall, our efforts during the hackathon paid off with an amazing product we can be proud of.

What we learned

Attending this hackathon was a great experience for us because we were able to learn a lot about machine learning algorithms, as well as frontend development and usage of different APIS. Firstly, our team had to conduct extensive research on sentiment analysis machine learning models, which helped us learn much about how supervised learning methods are implemented alongside neural networks. We also learned how to analyze data trends by creating graphs of the data and training accuracies of the model over time, and how to use these patterns to determine how the model can be improved. In addition to learning about machine learning, we all learned more about how to use different frontend languages, such as Javascript, HTML, CSS, and TypeScript, to develop an aesthetic user interface on a website. For example, we learned more about how to integrate forms onto our website, as well as creating transitions for different parts of our website such as user reviews. Finally, we learned about how to use a wide variety of different APIs and libraries in python. For instance, we learned a lot about how to use a speech-to-text library in python to transcribe the audio in a video. We also learned how to use the Gmail API, which allowed for us to send an email out to the user with a detailed report of the conversation they sent to us on the google form. These different aspects of our project helped teach us a lot about how to develop a full stack project, and it will ensure we continue to learn more and succeed in our future programming endeavors.

What's next for CrimiDect

We plan to modify CrimiDect so that it can transcribe longer conversations with more accuracy. We will do this by adding more test cases to further teach the algorithm and fine tune the deep learning model. We only had a day, so even though we were able to create a full and working tone detection, there are things that can be improved like supporting different file types and using a larger dataset. Next, we also want to create a server that can run continuously so that detectives all around the world can use our product whenever they want. This will create a larger scope for CrimiDect. Last of all, we want to provide a more detailed report of how malicious a transcript is from 1-10. This can be done by adding more training data with a scale of values, and this will make it possible for detectives to prioritize cases and know the true severity of criminals’ plans.

Share this project:

Updates