ApplianceSight

The Problem

Sighted people have a difficult time imagining the trouble that people who are blind face when trying to accomplish routine daily activities.

Right now, put on a blindfold.
Go to the kitchen.
Try to make popcorn in the microwave oven.

If you could manage to get to the kitchen and find the microwave oven, how would you figure out what buttons to press to cook your food? Dials and knobs that might offer tactile feedback are no more these days. Appliance manufactureres prefer sleek flat panel banks of buttons that offer no guidance to people who are blind.

So what would you do if it were movie night and you wanted to make popcorn for your date?

This is what a microwave oven looks like to a person who is visually impaired.

Engineering

Making sense of large amounts of information is a traditional application of data science. We have developed a tool to process large amounts of visual information and help people operate their kitchen appliances.

Our current prototype is composed of off-the-shelf retail components. It is essentially a Raspberry Pi with a camera that runs software to capture the image directly above the user's finger. Custom written computer vision algorithms will correct imperfections to the image. A neural network interprets the image and passes it to software to announce the key to the user.

Research & Analysis

User Research

Voice of the Customer

Image Processing

Data Workflow

Model Development

FCNN,CNN

5 Quadrant Image Centering

Hardware Evolution

Integration

Testing

Our partners have been instrumental in the requirement, development, feedback loop

Lighthouse for the Blind and Visually Impaired

East Bay Center for the Blind.

Noah Randolph testing ApplianceSight

Early testing at the LightHouse for the Blind and Visually Impaired in San Francisco

Recent testing at East Bay Center for the Blind

Team

We are a group of friends and data scientists who met at the University of California, Berkeley's Master in Information and Data Science program.

Amod Ghangurde

Anirudh Mittal

Noah Randolph

John Tabbone

Advisors

Joyce Shen

Alberto Todeschini

Tomorrow...

One of the key challenges as well as our biggest learning experience has been the difference in performance of our product when testing it in a controlled setting versus when testing it in the field. We hope to improve our product design based on key findings:

observed shaky hands while using the product - impacts image quality
starkly different lighting conditions - impacts image quality
angle of image capture (camera field of view and angle w.r.t keypad)
form factor issues

Roadmap

We plan to work on overcoming the challenges just described. We also need to generalize our model to work on different types of appliances. Leveraging the work that we have done so far, our team also envisions to develop partnerships with companies that provide audio/visual services to the blind.

Our vision is to finish providing utility in the kitchen, then apply the technology to other places.

Whether to help people live independent fulfilling lives or to update a 150 year old technology (Braille), many good things will come from this effort.

We need your help. We need partners, advisers, engineers, QA. We need resources. If you think you can help in any way, please reach out to us.

helping people who are blind gain more independence