Around 360 million people in the world are deaf or hearing-impaired. A majority of the population does not know sign language, which makes communication difficult with a hearing-impaired person. A large population of China is deaf, and so researches in the country have developed a system that uses a Kinect sensor to translate sign language into spoken word, and vice versa.
The system consists of an on-screen avatar that performs the signs to a hearing-impaired person. The person can then sign back to the system, and the Kinect sensor will track their hand movements and body language, and then attempt to interpret what the person is trying to say. The interpreted text is then read aloud via a computerized voice as well as displayed on screen.
This project was a result of a collaboration between Microsoft Research Connections, Chinese Academy of Sciences, Beijing Union University, and Microsoft Research Asia, each of which helped fine-tune the system to what you see in the video below.