SpellrTech

The working attachment!
It lives!
Original concept prototype
CAD design
Our mascot!

Inspiration

All of us can relate to the struggle of learning a new language. In evolutionary terms, humans only recently gained the ability to write, which means we have no biological mechanism compelling us to understand written language. This means an estimated 20% (or 1 in 5) kids have dyslexia - making the simple act of reading a book or deciphering a prescription label a laborious task.

Our team members have firsthand experience with dyslexia and working with dyslexic children, either through research or tutoring, motivating us to develop a discrete and effective tool to aid them in learning to spell. We recognize that a big pain point for dyslexics is that they need to spend 5x more effort for language tasks compared to others, justifying the need for frictionless assistive technology. With this in mind, we decided to combine our collective skills in design and coding to make a slick, effective pen add-on that helps anybody write with speech-to-text functionality! (and a game!)

And to emphasize, it's not just for those with dyslexia - it turns this same tech can also be used for learning any written language! All of our team members hail from different cultures - from Italy to China to the States - and we saw this project as a really cool chance to explore and appreciate something that unites us all as humans—language.

What it does

The core of SpellrTech is a pen gripper that contains a mini-display, accelerometer, and microcontroller. By holding the single button, users can speak into the onboard microphone, and the microcontroller will transcribe their speech into written language, while the screen gives a step-by-step tutorial of how the word is written - down to the exact pen stroke.

The attachment also uses our custom software, SpellrOS. Users navigate the interface by intuitively tilting their pen up or down, which includes relevant other information (dashboard, battery, etc.). We also included an onboard game - WordQuest - that uses the tilt feature to help kids get better at lexical automaticity (the ability to recognize common words quickly - something very important in assessing dyslexia).

The attachment then syncs up with our online platform, where the student's "Scriptor Profile" is gradually built. As students practice with the attachment, our platform uses some really neat clustering algorithms to identify phonetically similar words. By analyzing what words the kids struggle with most, we can develop a comprehensive profile for each user, allowing teachers or tutors personalized insights on their students. We can also make custom games and assessments tailored to each student's personalized needs.

All of this also applies to learning other languages! We included a small demo of simple Chinese character-writing tutorials, with plans to expand our offerings to other languages as well!

How we built it

Modern dyslexia assessments (Woodcock-Johnson, TOWRE, CTOPP, CVLT, etc.) cost literally thousands of dollars each, have to be refreshed every single year, and require a trained, in-person administrator. We built Spellr for HackTech in a weekend and use off-the-shelf parts that are extremely inexpensive.

The design for the 3D printed enclosure was made in Autodesk Fusion 360, and the microcontroller we used was a Teensy with an ATMEGA chip for its balance between modularity, size, and memory. We also included functionality in the software for a Bluetooth chip and use a cheap microphone and OLED-screen module to display the OS.

The online platform was made via Firebase as a backend, React for the frontend, with a variety of intermediate Python scripts to run our Phoneme-matching algorithm.

Challenges we ran into

We had additional plans to incorporate spelling detection: the ability to precisely track the exact pen strokes that a student makes. We had a variety of solutions in mind, such as IR cameras, optical mouse sensors, and IMU tracking, but were limited in scope in the materials we had on-hand. This remains our top priority as a future goal, as the ability to record pen strokes digitally would allow us to employ CV algorithms to detect, assess, and correct spelling deficiencies with extremely high precision.

Tracking fine motor skills would open a huge amount of possibilities for us, including the chance to expand into other subjects like math for individualized tutoring!

Accomplishments that we're proud of

Our attachment is even snappier and slicker than we realized, and navigating the OS with the accelerometer feels exactly right - even a kid could figure it out! We're also really proud of how much tech we managed to cram into the enclosure: especially for a first attempt.

What we learned

We learned a lot in terms of microcontroller programming and CAD design!

We also gained a lot of really valuable experience of doing formal research for our product - it took a long time for us to flesh out a product idea that could balance the needs of all of our relevant stakeholders (kids, teachers, parents, etc.)

What's next for SpellrTech

We have obvious next steps with stroke-detection via a camera-based or IMU-driven solution. We also want to really flesh out the online platform so it's super user-friendly and allows anyone to get insight into their learning journey. And of course, we want to add support for more languages so people can start learning better!

Potential!!! :)))

Ultimately, we are building SpellrTech seriosuly, and for the long term. The problem today is that many students (specifically, dyslexics) don’t receive the explicit phonetic-based teaching that they need for literacy. (1) Many districts fail to recognize the importance of phonetics and are subscribed to the flawed “Whole Language Approach”. (2) Teachers don’t have enough time for explicit and individual lessons for grade-school students struggling with reading and writing. (3) Seeking private help is inaccessible for many due to costs and location. These problems are highlighted in the statistics: a third of 4th graders, and 82% of black fourth graders, can't read.

Advancements in AI, particularly for speech and LLM, and robotic sensing technology now enable Spellr to be an all-in-one package: an accessible and personalized Orton-Gillingham tutor for every student.

At the core, the SpellrTech tutor needs to be multisensory. Those who are struggling with reading and writing need to connect the dots and receive reinforcement through audio and images. They also need to sound new words out themselves, use their motor skills, like writing, or air writing, to reinforce their spelling.

For example, large language models now enable the creation of completely custom “level books” and “decodable” short stories. That is, completely personalized stories that match the general skill of students and train/reinforce specific phonemes that they recently learned. Image generation models allow on-demand creation of visual reinforcement for writing and reading. Text-to-speech models are becoming more realistic, allowing a tutor to sound exactly like a human. Speech recognition models are now able to accurately identify words spoken, and also the specific phonemes a student says when sounding out a word / reading, allowing for automated reading assessments and intervention. On the hardware end, sensing technologies like IMU and camera tracking make it possible to track the writing of a student and to automate tracking of “air writing”, a common technique for language learners. The scale of embedded systems now allows spell-tracking technology to be fit on a pen, or a computer webcam.

Our goal is to digitalize the Orton-Gillingham experience - essentially, a multimodal, multisensory, phonetic-based teaching methodology for kids with dyslexia. While we are designing the curriculum for the computer screen, connecting to the SpellrTech pen is crucial for this vision because (1) the stroke tracking for fine motor skills in handwriting is one pillar of multisensory approach, and (2) the pen intimate, frictionless, and available help at the fingertips of those who need it, unlike any other existing technology/method.

Currently, applying the complete Orton-Gillingham experience to every student is impractical and unscalable, which highlights a flaw in the existing educational system—it lacks the capability for individualized, multisensory instruction. However, with SpellrTech, and recent waves of tech innovation, this approach becomes a feasible reality for every student!