Sign language generation on smartphones

(this project involves a generation component by video concatenation as a class project for my Introduction to NLG class at FaMAF)

In this project, we seek to build a limited domain Android application with a limited domain automatic speech recognition system and a domain specific language generation targeting sign language.

For a first pass, we want to focus on the use of video concatenation as a (suboptimal) way of generating sign language and medical visits as the domain.

The target language is the Argentina sign language or LSA (Lengua de Señas Argentina). The input language is Spanish.

Tasks

• Crawl http://www.manosquehablan.com.ar/diccionario/ (Sergio, done)
• Find grammar resource for LSA (either book, video or native signer informant) (Sergio )
  • LSA - ASAM Jose Antonio Terry (Tomo 1, 2, 3 y 4)(pdf)
  • Diccionario Lengua de Señas Argentina Español (pdf)
  • Material de Estudio - Comunicacion Basica en LSA (pdf)
  • http://www.youtube.com/watch?v=CDvFC6ddV64
  • http://www.youtube.com/watch?v=SXEDbn2kaMc

• Acquire a medical phrasebook (DrDub, done)
  • http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_4073230 (emergency phrasebooks in 36 languages)
    • Spanish one (57 phrases): http://www.dh.gov.uk/prod_consum_dh/groups/dh_digitalassets/@dh/@en/documents/digitalasset/dh_4073466.pdf
    • ASCII version in /proj/lsa

• Transform the doctor phrases into predicates for NLG (Sergio)

• Write paraphrases for the doctor phrases (DrDub)
• Record doctor phrases and train domain dependent ASR

• Find missing vocabulary with respect to signing dictionary
• Record missing vocabulary

Development

1. Re-code videos in 3gp format (DrDub, done)
   1. ffmpeg -i farmacia1.swf -s qcif -r 25 -ab 32 farmacia1.3gp
   2. http://developer.android.com/guide/appendix/media-formats.html
   3. http://goinggnu.wordpress.com/2007/02/13/convert-avi-to-3gp-using-ffmpeg/
2. Make simple Android app that shows the video (DrDub, done)
   1. see /proj/wip/sonajero-0.2.zip
   2. http://developer.android.com/resources/samples/ApiDemos/src/com/example/android/apis/media/VideoViewDemo.html
3. Show video + camera view
   1. http://developer.android.com/reference/android/hardware/Camera.html

1. Show video blended with camera view (with transparencies)

Links

• LSA
  • (wikipedia ES) Lengua de señas argentina
  • Manos Que Hablan/Alfabetos/Argentina
  • Confederación Argentina de sordomudos
  • Centro de Recursos para Personas Sordas (Córdoba)
  • Diccionario animado de LSA
  • Desarrollan avatar en 3D capaz de comunicarse en lengua de signos española

• Mobile sign language
  • MobileASL Project Brings American Sign Language to Cell Phones
  • CMU Sphinx ASR
  • Pocketsphinx for mobile
  • PocketSphinx on android via the NDK (git repository doesn't exist)
  • PocketSphinxDemoApp cloned here
  • PocketSphinx Android-ndk discussion
  • PocketSphinx Android-ndk Stack Overflow discussion

• Sign language Generation
  • ASL NLG
  • GENERATING ASL CLASSIFIER PREDICATES FOR ENGLISH-TO-ASL MT
  • A Survey and Critique of ASL NLG y Machine Translation System
  • Grammar Development for BSL Avatar-Based synthesis
  • Sign Language Generation in an ALE HPSG
  • Evaluation of ASL Generation by Native ASL Signers
  • GANAS: Generador Automático del Lenguaje de Signos
  • Produção de Textos Paralelos em Língua Portuguesa e uma Interlíngua de LIBRAS (SL and TS)
  • PROJETO PUL∅ (sistema de tradução automática unidirecional de uma língua oral-auditiva, o português, para a representação linear de uma língua gestual-visual)
  • SignSynth