PepperPay

The fun way to buy goods from your favorite retailers.

Comment 1

Inspiration

A few nights ago, as Dave was wondering home after a late-night hack session at the office, he wanted to stop by Walgreens to pick up a fresh tube of toothpaste. He picked up one single item from their easily laid out shelves and got in line. His stay was only beginning. After two arguments between patrons and cashiers and a very slow person, Dave finally walked out of the store 30 minutes after he arrived. He knew there had to be a better way to purchase simple goods. It's 2016 for god-sakes!

For years, stores such as Target, Banana Republic, and CVS have been pouring millions of dollars into research about ways to better layout their stores. However, there is one common problem that has yet to be solved: the lag experienced when it comes time to check out.

What it does

PepperPay makes checking out of your favorite store easier. It starts at the store, by being asked if you need any help with your store visit. It then allows you to buy the goods you've picked out, without having to deal with a cashier. All you have to do is hold up each item in front of the camera. It will then use image recognition (courtesy of IBM Watson) to figure out which item it is. It will then match that against a database to let you know the price. Without even taking out your wallet, you can then press a button to log into PayPal and pay. This process reduces the average checkout time by over 50%. Built with love by Team Croissant.

How we built it

We started by designing a simple flow for checking out different items. We then built a customized web page for the tablet display on the Pepper robot (courtesy of SoftBank). After learning the different gesture and other functionality using Choregraphe, we decided to go with the JavaScript wrapper for the Python API. This enabled us to quickly iterate on different robot control ideas. While that was being worked on, Adam tapped into the Watson Image Recognition API for taking the pictures from Pepper and converting them to specific items that we could put a price on. Lastly, we integrated PayPal via Braintree into the web page and back end, for completing the transaction.

Challenges we ran into

One major challenge was the fact that the team had little to no robotics experience before this. We had to interface with SoftBank representatives who quickly and efficiently helped us learn the tools for the job. A big challenge with the tablet web page was that is was opened as a single file in the tablet web browser. This presented an issue with PayPal, which requires a hosted web page in order to open up the pop-up payment checkout window. We ended up adding a simple button on the front end to bypass this issue.

Accomplishments that we're proud of

We're proud of what we've managed to make this robot do in such a short time, and of having worked with so many cool APIs in the process. Getting the arms to move and the robot to roll forward, backward, and in circles was the "ah ha" moment that really got us excited.

What we learned

We now know the difference between "yaw", "pitch", and "roll"! We also know about the challenges behind getting a robot to do what you want. The latest robotics technology today is kin to the latest mobile tech of ~10 years ago. It is only a matter of time before it improves and becomes omnipresent.

What's next for PepperPay

We don't know too much about the retail game, but we imagine a future where something like this changes how we shop. We invite any and all to take a look at our code on github and see where they can take it!

Share this project:

Updates