Sign language generation on smartphones

(this project involves a generation component by video concatenation as a class project for my Introduction to NLG class at FaMAF)

In this project, we seek to build a limited domain Android application with a limited domain automatic speech recognition system and a domain specific language generation targeting sign language.

For a first pass, we want to focus on the use of video concatenation as a (suboptimal) way of generating sign language and medical visits as the domain.

The target language is the Argentina sign language or LSA (Lengua de Señas Argentina). The input language is Spanish.


  • Transform the doctor phrases into predicates for NLG (Sergio)
  • Write paraphrases for the doctor phrases (DrDub)
  • Record doctor phrases and train domain dependent ASR
  • Find missing vocabulary with respect to signing dictionary
  • Record missing vocabulary


  1. Re-code videos in 3gp format (DrDub, done)
    1. ffmpeg -i farmacia1.swf -s qcif -r 25 -ab 32 farmacia1.3gp
    2. http://developer.android.com/guide/appendix/media-formats.html
    3. http://goinggnu.wordpress.com/2007/02/13/convert-avi-to-3gp-using-ffmpeg/
  2. Make simple Android app that shows the video (DrDub, done)
    1. see /proj/wip/sonajero-0.2.zip
    2. http://developer.android.com/resources/samples/ApiDemos/src/com/example/android/apis/media/VideoViewDemo.html
  3. Show video + camera view
    1. http://developer.android.com/reference/android/hardware/Camera.html
  1. Show video blended with camera view (with transparencies)