Using Convolutional Neural Networks for Fingerspelling Sign Recognition in Brazilian Sign Language

• Douglas F. L. Lima UFPB
• Armando S. Salvador Neto UFPB
• Ewerton N. Santos UFPB
• Tiago Maritan U. Araújo UFPB
• Thais G. do Rêgo UFPB

Resumo

Deaf people communicate naturally using sign languages, and because of this, they have difficulty in communicating using oral or written languages. To minimize this problem, one alternative is to use machine translation systems from sign languages into oral languages, especially for scenarios where human interpreters are not viable or unavailable. In this work, we address this problem by proposing a solution for fingerspelling recognition in Brazilian Sign Language (Libras) using Convolutional Neural Networks. The system uses a 224000 images dataset created by our team, which represents the letters of the Libras alphabet signed by 12 people in different backgrounds, body arm, hand positions, and lighting patterns. The results show that the solution had an average accuracy of approximately 99% in a dependent person scenario, and had an average accuracy of 71% in an independent person scenario. This type of solution can be used together with machine translators from Brazilian Portuguese to Libras, to assist in the communication of Brazilian deaf people.