Isolated Sign Language Recognition in LIBRAS

• Marcelo M. Delucis PUCRS
• Otávio Parraga PUCRS
• Rodrigo C. Barros PUCRS
• Lucas S. Kupssinskü PUCRS

Resumo

Hearing loss affects a significant part of the world population. Sign Language Recognition is the task of recognizing isolated signs in videos and has the potential to ease communication between Deaf and non-Deaf people. Data are vital to creating better Sign Language Recognition models. LIBRAS, the Brazilian Sign Language, still lags behind in terms of data availability in comparison to other Sign Languages, such as American and Russian. Our study aims to bridge this gap by introducing a novel LIBRAS dataset for Isolated Sign Language Recognition, MALTA-LIBRAS. This dataset is assembled from open online sources and can be used as a more diverse and representative test set than the previous available LIBRAS dataset, MINDS. We perform several experiments with three transformer-based video classification models for this task: VideoMAE, TimeSformer, and ViViT. Our findings show that action recognition pretraining substantially improves generalization beyond fine-tuning data distribution. Additionally, we identify a set of data augmentation strategies that further increase the model’s generalization to new conditions. In contrast to what could be expected, transferring knowledge from other sign languages does not yield significant improvements for LIBRAS.