Inspiration 💡

The Coronavirus outbreak has caused major disruptions to our daily lives. Currently, the fear of the third wave has imposed lockdown in several countries. Changes in social dynamics due to local restrictions immensely impacted human behavior and are making humans lethargic. According to nimh.nih.gov, the same has resulted in an over 50% increase in the number of people affected by Attention Deficient Hyperactivity Disorder (ADHD) which is a matter of great concern! The most affected ones are specially-abled people as well as children.

The current scenario in fact acts as a catalyst & results in low-esteem, lethargy, limited attention span, hyperactivity, and even can cause paralysis.

We believe that with the power of AI, this can be solved if proceeded creatively. Thus we made EYEBalliN!

What it does 🤔

EYEBalliN' is a single-player TPF (Third person View) game based on Tensorflow.js, where users can aim and shoot the basketball into the hoops with the blink of an eye! Using cutting-edge technologies, we are bringing the fun of playing basketball 🏀 to your doorstep. You just need to wait and target a hit at the proper angle and wink! Boom, you will get the shot! Also, for people who just want to play the game for fun, there is an option to hit spacebar, and the ball shoots. 🎯 And also not to mention, we also added some Easter Eggs inside this game! 😉

Just visit keep-balling-with.us on your PC 💻 & slam!

How we built it ⚙️

EYEBalliN' is crafted with ❤️. EYEBalliN' is primarily a Javascript-based Webapp, running on Tensorflow.js which segments the eyes from the face under the hood and tracks the user's face to detect winks and log the current session segmented upon various factors. In the beginning, the video-feed is processed via TFjs's face-API for facial detection.

Then we sample the stream into distinct frames to extract few metadatas like — Facial Landmarks & Expressions. Furthermore, on those sampled frames, we apply OpenCV.js to strip the eyes and then process the image after soothing them to remove noise via bilateral filters. Then we find the contours and centroid of the eye clips. All of the above caters in wink-detection! Feel free to refer to this for more info.

We are using a few 3rd party API's like chart.js for rendering the system usage plots processed via the chrome.processes API. All of the computations are done on the client-side itself. The model loads for once & is stored in localstorage which enables it to work offline too & ensures 100% GDPR compatibility offering seamless user privacy! Adding to the above, the previously mentioned facial expressions are also used to trigger songs via the Spotify-API. The application is deployed on a free tier of Netlify.

Challenges we ran into 😤

A lot! As I previously mentioned, the whole execution was done from scratch, even the advent of idea in our mind literally came during the opening ceremony from Sandipan. Initially, we were facing some issues while training the model on our system, including underfitting errors as we had to reduce the parameters to optimize it so that it can run seamlessly on low-end devices. Frozen weights of TFjs caused a heck of a mess for us and GPU became the main factor for the project. Also, it was a bit difficult for us to collaborate in a virtual setting but we somehow managed to finish the project on time.

Research 📚

Development of a Fast and Robust Gaze Tracking System for Game Applications, arXiv[cs.CV], 2020 : https://arxiv.org/ftp/arxiv/papers/2105/2105.02460.pdf
Efficiency in Real-time Webcam Gaze Tracking, arXiv[cs.CV] 2 Sep 2020 : https://arxiv.org/pdf/2009.01270.pdf

♣ Datasets :- Recompiled from several sources.

♣ Articles :-

ADHD with COVID-19, SagePub : https://bit.ly/3AYEX6X
Impact of COVID-19 and lockdown on mental health of children and adolescents, ncbi.nlm.nih.gov : https://bit.ly/2XH5gA1
Impact of COVID-19 on persons with disabilities in India, Hindustantimes : https://bit.ly/3DhqTrh

CREDITS

Design Resources : Freepik
Icons : Icons8

Takeways

Accomplishments that we're proud of ✨

We are proud of finishing the project on time which seemed like a tough task as we started working on it quite late due to other commitments and were also able to add most of the features that we envisioned for the app during ideation. Moreover, we learned a lot about new web technologies and libraries that we could incorporate into our project to meet our unique needs. And as always, working overnight was pretty fun! :)

What we learned 🙌

A lot of things, both summed up in technical & non-technical sides. For the technical side, we got to learn so much while building game engines which are around 80% Maths & Physics 😛. We also gave our level best to make the UI/UX look solid which helped us learn more about different design-centric approaches! Not to mention, Stackoverflow was the gem for us while we're troubleshooting some complicated issues late-night.

What's next for EYEBalliN 🚀

Finish the unfinished ones, which include adding more scenarios to the playground followed by improvements in the custom model so that the same can be run on low-spec systems. The code might be refactored for adding responsiveness & support for mobile devices. Apart from fine-tuning the project, we're also planning to add multiplayer support to the same so that people can join in rooms to compete with each other. Overall, we hope that one day this project can be widely used globally to redefine the existing & remove the backlogs.

Note — Spotify credentials have been revoked! To integrate, run the project on your local with your own credentials which can be found at spotify.com/account. Also, please make sure that your spec has at least 8gb's of RAM & standard VGA memory to have a seamless experience!