Skim | Devpost

iOS home screen
iOS highlight
iOS text
Android home screen
Android highlight
The future

Inspiration

As students growing up in the 21st century, we have our fair share of experience with both digital and physical school resources. However, the transition between the two is often frustrating due to the lack of tools offered on the physical side, like finding keywords or copy-pasting passages. Our members have (an embarrassing number of times) tried to press Ctrl-F over an AP Euro textbook to no avail. We wanted to use the technology available in 2017 to close the gap between traditional studying and smart learning.

What it does

Skim is a mobile application available on Android and iOS that takes a picture of text as found in something like a textbook, legal document, or newspaper and utilizes Microsoft's Computer Vision API to recognize the text and allow further manipulation. Users can search for a keyword in the text and the app will highlight its position in the image. Furthermore, users can access the text conversion to read and copy passages.

How we built it

The application was built on the respective mobile platforms: Xcode for iOS and Android Studio for Android (along with a healthy dose of Vim). It calls Microsoft's Computer Vision API once a picture is taken or selected and stores the resultant JSON information with the text content and positions of each word. We then developed algorithms using that information to calculate and highlight the keyword's position on the page. We used Adobe Illustrator to mock-up the UI and create all the icons.

Challenges we ran into

Skim represents some of our team members' first forays into mobile app development and the Android team faced a huge learning curve as none of them had extensive experience developing on Android and were given a mere day and a half to both understand and make a relatively polished prototype with Android Studio. For the prototype, we decided to use Flutter, however its immaturity caused extra work due to the lack of open source libraries. Our iOS developer also faced new challenges in learning to integrate a camera and manipulating images within the app.

Accomplishments that we're proud of

We are very excited about our use of Microsoft's immensely powerful Cognitive Services with their computer vision API. It is the first time any of us have used such cutting-edge resources in a useful application that we would use everyday. We are also satisfied of our product design in creating an easy and aesthetically-pleasing user experience. We decided on the name "Skim" because it reflects our core design philosophy of a lightweight and "fat-free" software experience. We're proud to live up to that name with our application.

What we learned

Each member gained invaluable experience in application development, from integrating Microsoft APIs to creating a clean and polished UI. The team made strides in graphic design with both designing and integrating the flat design logo and app icons. Our junior programmers made their first Android application and implemented an impressive amount of functionality within the time constraint. We also all learned a lot about taking pictures and working with their various aspects, from compressing before sending to drawing shapes on them.

What's next for Skim

We see a very optimistic future for Skim as we further blur the line between the physical and digital world. We already have ideas for more features like PC syncing and a web app so we can bring this tech to our more powerful and productive laptops/desktops. As it becomes more mainstream, we look forward to developing Skim AR for augmented/mixed-reality devices like the revolutionary Microsoft HoloLens. Using the sensors and processing power on-board AR headsets, Skim AR will have the same functionality as Skim in real-time and project information intuitively onto the real environment. We can see Skim and Skim AR being used to help everyone from middle-schoolers studying for their next quiz to the elderly who need help reading small text on a label. Our team understands that we live in a very physical world, and will need all the help we can get for a long while to come. With that in mind, Skim can help us streamline our lives, one word at a time.