According to Professor Brian Mak, Associate Professor in HKUST’s Department of Computer Science and Engineering, a recent survey indicated that Hong Kong is home to around 50,000 people who are either totally deaf or are hearing-impaired. However, the kind of support this community requires is not currently available.
“The number of professional sign language interpreters [in Hong Kong] is low – only about 54 in total,” Professor Mak explained.
For their final year project, three Computer Science and Engineering undergraduates decided to use the skills they had developed at HKUST to tackle this problem. The app developed by Ken Ka-wai Lai, Mary Ming-fong Leung and Kelvin Wai-chiu Yung translates Chinese sentences keyed into a phone into video sequences illustrating how to sign the words.
Professor Mak sees the app being of great use to those training to be interpreters, as well as to the general public. “This project is a good example of our passion to serve,” he said.
One of the main challenges the students faced arose from the differing sentence structures in sign language and Chinese. “In sign language, adjectives, adverbs, numbers and question words are normally placed after the noun, which is not the case in Chinese,” Lai pointed out.
The team resolved this issue by designing a new sentence segmenting algorithm. “We make use of the FFmpeg software to carry out video synthesis – putting the various signs performed on video together to form a complete sentence,” Yung explained.
As she has a good command of sign language, Mary Leung became the ‘signer’ in the video clips. “It was a very memorable experience for me – I recorded over 1,700 words and expressions in sign language over a span of half a year,” she said.
For more information, please click here.
