Orator.ai

Project Landing Page
Dashboard with Dropdown
Upload Page
Team Photo!!

Inspiration💡

The inspiration behind Orator.ai is deeply rooted in our ability to learn and our fascination with self-awareness/self-reflection. At one point or another, we've all felt the need to learn a skill but never knew where to start. The ability of public speaking isn't just a skill we build for ourselves but is something that can motivate and influence those around us. But oftentimes, it may be hard to find what's right for us in self-learning and growing through self-reflection. 🤔

We embarked on a journey to create Orator.ai with a vision of finding that perfect fit, by harnessing technology and innovation to increase our abilities to learn and remove barriers to learning. 📷

What it does📖

Orator.ai isn't just an educational public speaking app, it's a way to find information for you. Orator.ai brings you from start to finish in your learning journey, providing feedback and analytics as you upload videos to improve all aspects of your public speaking skills.

Get started on the Landing Page and sign up for a free account, with user information encrypted by JWT.
View and interact with the dashboard or press upload to provide a new video for analysis.
Once uploaded, the video is compressed framewise and bitwise by ffmpeg and transcribed by using Google Cloud's Speech-to-Text.
Then with API calls to VertexAI Vision and Gemini in conjunction with LangChain for concurrency optimization and MongoDB for data storage, highly researched and engineered prompts obtain key points, feedback, and personalized course outlines.
This data is fed into the YouTube API to find the most suitable educational video.
Users can view their results, and past recordings and edit previously created lesson plans to optimize their studying curriculum on their dashboard.

Orator.ai reduces the time it takes to find something that's relevant to each person and eliminates the barrier of entry in learning new skills efficiently.

How we built it 🛠️

We built Orator.ai with a variety of amazing technologies:

FrontEnd

Next.js: The building blocks for the front end 🧱
Vercel: Hosting and tracking deployments for the site 💻
TailwindCSS, SASS, Framer Motion, Three.js: creating a stylish, animated, and user-friendly UI/UX ✨
GoDaddy: for the amazing domain name🤩

BackEnd

Flask.py: Hosting backend server and API calls
Modal.com: Used for serverless backend deployment
LangChain: Employed for concurrency and optimization of API calls 💨
MongoDB: Storing user data, prompted feedback, and course data 📊
JWT: For encrypting users' data 🔒 And most importantly, the suite of Google Developer tools we used:
VertexAI: Leveraging VertexAI Vision to analyze communication gestures 👋
Gemini API: Prompting for feedback and generated course content 🤖
Google Cloud Speech-to-Text: Transcribing Speech-to-Text (STT) 🎤
Youtube Search API: Querying suitable educational videos based on generated content 🎥

Challenges we ran into🚧

Our journey in creating a scalable and efficient application was not without our trials and tribulations. With the large amount of API calls made to VertexAI and Gemini, hallucinations had to be minimized through careful prompt engineering and relentless testing. Additionally, utilizing the variety of vital frameworks and libraries proved to be a challenge in connecting and testing modules.

Accomplishments that we're proud of🏆

Standing together as a cohesive team with a wide array of different experiences, we were delighted that we were able to bring together an app that made the most of our skill sets. Using Google's Developer Tools, we were able to link together processes that we had thought would be infeasible at the start and finished our application in under 24 hours. The ability for different tools to function together to reduce inefficiencies and increase scalability allowed us to maintain a more diverse and flexible system with exchangeable and reusable modules.

What we learned📚

The journey to get to our final idea was a long-winded one. Starting with some simple piece of speech-to-text and building slowly upon it led us to our creative and impactful solution. We've learned that innovation is oftentimes built upon the most unexpected inspiration. An idea we may first perceive as laughable may end up being one of the most impactful.

What's next for Orator.ai🚀

RAG and Data Analysis 📈 Orator.ai isn't just done yet. Rather, we're just beginning. With a future-focused mindset in the interest of people seeking to become Oratorized, we want to bring more accurate and informative data to our users by implementing Retrieval-Augmented-Generation (RAG) and building out our features for Data Analysis.

Building a community 👥 Speaking is inherently a social activity and you're not alone. We want you to be able to connect with a community whether through a feature, messages, or a shared experience so that you don't feel alone in your journey. Orator.ai is more than just a learning platform; our goal is to become a platform for shared knowledge and experiences.

Other Applications Orator.ai is boundless. With different engineered prompts and slight changes in the codebase behind Orator.ai, users can find educational content for all sorts of purposes, especially those with Computer Vision applications (i.e. sports). The future of Orator.ai is all about pushing boundaries and providing users with a way to interact with their physical and auditory learning in a whole new way.