Padmanabhan KRISHNAMURTHY (Paddy)’s sister worked with an NGO in India that helps people with visual impairment. He observed that people with any kind of impairments face immense difficulties, and wanted to do something to alleviate their problems. This inspired him and his teammate Amrutavarsh Sanganabasappa KINAGI (Amrut) to invent a lip-reader named “Helen”, which has just won the top prize in the 2019 HKUST President’s Cup that encourages undergraduate students to showcase creativity, innovation, and excellence.
Named after Helen KELLER, the renowned American author and educator who overcame the adversity of being blind and deaf to become one of the 20th century's leading humanitarians, the lip-reading device is expected to benefit millions of people when it’s well developed. According to the World Health Organization, there are 466 million people globally with some form of disabling hearing loss, and this number is forecasted to grow up to 900 million by 2050.
Helen is the first-ever wearable device that performs lip-reading. Currently, hearing aids only use audio, and hence do not work well in noisy surroundings. And those that do perform well with noisy backgrounds are quite expensive and aren’t widely produced. With Helen, hearing-impaired individuals would no longer have to rely on audio information, or expensive hearing aids, as Helen would enable them to receive information through an entirely new visual dimension. It is an AI-based lip-reader that utilizes a camera to stream video of a speaker to a remote device that runs LipNet, a model that maps a variable-length sequence of video frames to text co-developed by the University of Oxford, DeepMind, and CIFAR.
Paddy is a Computer Science student and Amrut is majoring in Computer Science and Mathematics, and both are going into their third year of studies this September. They owe a debt of gratitude to their project advisor, Prof. Brian MAK, for helping them turn their ideas into an award-winning innovation. “We are very grateful to Prof. Mak for pointing us in the right direction throughout the duration of the competition, and critically, connecting us with his former students who worked on lip-reading for further guidance and tips,” they say.
While Helen’s first use is to aid those with hearing problems to understand speech, it has other potential specialist uses too. Court clerks and legal professionals can deploy Helen for court cases; it can also assist police investigations in determining what people are saying in CCTV footage, they explained.
The two innovators are now busy improving the device. “Adding the ability for Helen to lip-read while facing a person from the side would be one of our top priorities,” they remark. Another adjustment they wish to make is expanding the different lighting conditions where Helen could work effectively. At this moment, the lip-reader’s optimal lighting condition is well-lit environments. Increasing the angles it can lip-read from and improving usability in darker conditions would vastly enhance Helen’s effectiveness. Paddy and Amrut also hope to make the lip-reader even more compact so that it could be clipped to the wrists.
They have begun procuring new circuit boards with microphones on them to add the audio function to Helen, and are also in the process of re-implementing the AI for better results. Right now Helen is confined to English since that’s the only dataset currently loaded but the inventors emphasize adding new language datasets is very important. Having successfully obtained the licenses to use the LRS (Lip Reading Sentences) and LRW (Lip Reading in the Wild) datasets developed by the BBC/Oxford, the pair now plans to use them once they have reworked the AI to support the new data and refined the hardware to use microphones.
The team is also well aware of HKUST graduates’ long record of establishing inspirational start-ups and is already proceeding with plans to launch Helen as a commercial product with the University assisting them in matters concerning intellectual property. They have wasted no time in tapping into the HKUST Entrepreneurship Center and its alumni network to take their project to the next level.
Paddy and Amrut encourage their fellow students to take part in a range of entrepreneurship competitions organized by the University to materialize their ideas into real projects. “We have to thank the President’s Cup for enabling this to happen, we would have never embarked on this project if we didn’t have access to both the time and resources this competition allows,” they say.
The masterminds behind Helen are demonstrating how to skilfully transform research and innovation into a device that can greatly impact the lives of millions of people worldwide, fulfilling the true purpose of tertiary education in the 21st century.
