CopyCat is designed both as a platform to collect gesture data for our ASL recognition system and as a practical application which helps deaf children develop working memory and language skills while they play the game.
Please view the video below
The system uses a video camera and wrist mounted accelerometers as the primary sensors. In CopyCat, the children use ASL to communicate to the heroine of the game, Iris the cat. For example, the child will sign to Iris, "ALLIGATOR ON CHAIR" (glossed from ASL). If the child signs poorly, Iris looks puzzled, and the child is encouraged to attempt the phrase again. If the child signs clearly, Iris "poofs" the villain and continues on her way. If the child cannot remember the correct phrase to direct Iris, she can click on a "help" button. The system then shows a short video with a signer demonstrating the correct ASL phrase. The child can then mimic the signer to communicate with Iris. This is similar to the help a mother provides a child when the child is not sure what to say in a sitiuation.
Gesture-based interaction expands the possibilities for deaf educational technology by allowing signing children to interact with the computer using their gesture-based language. An initial goal of the system, suggested by our partners at the Atlanta Area School for the Deaf, is to elicit phrases which involve three and four signs from children who normally sign in phrases with one or two signs. This task encourages more complex sign construction and helps develop working memory for language.
The initial version of CopyCat had used a "Wizard of Oz" approach where an interpreter simulates the computer recognizer. This method allows research into the development of an appropriate game interface as well as data collection to train our hidden Markov model (HMM) based ASL recognition system. The current version utilizes computer reecognition to determine the corectness of the children's signing. The recognizer is accurate 85% of the time in correctly assesing the correctness of the children's signing. This level of accuracy did not cause the children any undue frustration and also alowed the children to make significant gains on measures of working memory, language comprehension and language expression.