Machine learning approaches for efficient recognition of Brazilian Sign Language

• Letícia M. G. de Morais UFERSA
• Welizangela M. Almeida UERN
• Rosana C. B. Rego UFERSA http://orcid.org/0000-0001-5997-1221

DOI: https://doi.org/10.5753/sbsi.2025.245970

Resumo

Context: The Brazilian Sign Language (LIBRAS) is the primary language for the Brazilian Deaf community. However, the lack of LIBRAS interpreters presents challenges for the inclusion of individuals with hearing impairments in society. Additionally, learning a sign language can be as difficult for hearing individuals as learning a foreign language. Problem: There is no official translator between Brazilian Portuguese and LIBRAS, making communication between hearing and Deaf individuals challenging. Solution: This study presents machine learning algorithms that can recognize 44 LIBRAS signs and translate them in real time to Brazilian Portuguese. IS Theory: This work is grounded in the Diffusion of Innovations Theory, focusing on how new technologies are adopted within communities. The proposed LIBRAS recognition system serves as an innovative solution to enhance communication between hearing and Deaf individuals. Method: This research adopts an experimental approach with quantitative and descriptive analyses. Data were collected from three different datasets, representing a total of 44 distinct LIBRAS signs. MediaPipe Hands was used to extract the spatial coordinates of hand movements. Then, some machine learning models were employed to recognize the signs, comparing their performance based on accuracy, F1 Score, precision, and recall metrics. Summary of Results: The results indicated that the ExtraTree algorithm was the best choice for the 44 signs dataset, achieving an accuracy of 98.01%, with an F1 Score of 98.01%, precision of 98.02%, and recall of 98.01%. Contributions and Impacts in IS Area: This paper contributes a low-cost approach for sign language recognition that requires only a camera to detect hands and recognize signs.

Palavras-chave: Artificial Intelligence, Neural Networks, Machine Learning, Brazilian Sign Language

Referências

Sílvia G. M. Almeida, Tamires M. Rezende, Gabriela T. B. Almeida, Andreia C. R Toffolo, and Frederico G. Guimarães. 2019. MINDS-Libras Dataset. DOI: 10.5281/zenodo.2667329

Silvia Grasiella Moreira Almeida, Tamires Martins Rezende, Andreia Chagas Rocha Toffolo, and Cristiano Leite de Castro. 2019. Libras-10 Dataset. DOI: 10.5281/zenodo.3229958

N. F. Andrade Junior, A. A. B. F. Pinto, W. M. Almeida, and Rosana C. B. Rego. 2024. Brazilian Sign Language Translation: AI for the Inclusion of Deaf People. In Anais do V Congresso Brasileiro Interdisciplinar em Ciência e Tecnologia. V Congresso Brasileiro Interdisciplinar em Ciência e Tecnologia.

Clodis Boscarioli, Renata M. Araujo, and Rita Suzana P. Maciel. 2017. I GranDSI-BR – Grand Research Challenges in Information Systems in Brazil 2016-2026. Brazilian Computer Society (SBC), Brazil. 184 pages.

KevinW. Bowyer, Nitesh V. Chawla, Lawrence O. Hall, andW. Philip Kegelmeyer. 2011. SMOTE: Synthetic Minority Over-sampling Technique. CoRR abs/1106.1813 (2011). arXiv:1106.1813 [link]

Eros Caiafa, Allan Basilio, Amaro de Lima, and Gabriel Araujo. 2023. Interpretação de gestos de Libras usando K-means e Random Forest. In XLI Simpósio Brasileiro De Telecomunicações E Processamento De Sinais (SBrT2023). Biblioteca Da SBrT. DOI: 10.14209/sbrt.2023.1570923868

AL Cavalcante Carneiro, L Brito Silva, and DH Pinheiro Salvadeo. 2021. Efficient sign language recognition system and dataset creation method based on deep learning and image processing. In Thirteenth International Conference on Digital Image Processing (ICDIP 2021), Vol. 11878. SPIE, 11–19.

Rodrigo A de Freitas Vieira and Clodoaldo A de Moraes Lima. 2018. Channel selection for EEG-based biometric recognition. In Proceedings of the XIV Brazilian Symposium on Information Systems. 1–8.

Herdney Souza dos Santos, Leila Fabiola Ferreira, and Poliana Gonçalves Leite Alves. 2019. Luva Tradutora da Língua Brasileira de Sinais. Trabalho de Conclusão de Curso (Bacharelado em Engenharia da Computação) – Centro Universitário Internacional Uninter. Orientador: Prof. Me. Charles Way Hun Fung.

Google AI Edge. [n. d.]. Guia de detecção de pontos de referência manuais. [link]. Acesso em: 29 de setembro de 2024.

Sam Fletcher and Md Zahidul Islam. 2019. Decision tree classification with differential privacy: A survey. ACM Computing Surveys (CSUR) 52, 4 (2019), 1–33.

Karina Freitas. 2021. Dia Internacional da Linguagem de Sinais procura promover a inclusão de pessoas surdas.

Kiana Hajebi, Yasin Abbasi-Yadkori, Hossein Shahbazi, and Hong Zhang. 2011. Fast approximate nearest-neighbor search with k-nearest neighbor graph. In Twenty-Second International Joint Conference on Artificial Intelligence.

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems 30 (2017).

Oliver Kramer and Oliver Kramer. 2013. K-nearest neighbors. Dimensionality reduction with unsupervised nearest neighbors (2013), 13–23.

D.F.L. Lima, A.S. Salvador Neto, E.N. Santos, T.M.U. Araujo, and T.G. Do Rêgo. 2019. Using convolutional neural networks for fingerspelling sign recognition in Brazilian sign language. In Proceedings of the 25th Brazillian Symposium on Multimedia and the Web, WebMedia 2019.

Yanli Liu, YourongWang, and Jian Zhang. 2012. New machine learning algorithm: Random forest. In Information Computing and Applications: Third International Conference, ICICA 2012, Chengde, China, September 14-16, 2012. Proceedings 3. Springer, 246–252.

Saulo Martiello Mastelini, Felipe Kenji Nakano, Celine Vens, André Carlos Ponce de Leon Ferreira, et al. 2022. Online extra trees regressor. IEEE Transactions on Neural Networks and Learning Systems 34, 10 (2022), 6755–6767.

World Federation of the Deaf (WFD). 24AD. UN Enable - Promoting the rights of Persons with Disabilities - Contribution by WFD. [link]

Aakash Parmar, Rakesh Katariya, and Vatsal Patel. 2019. A review on random forest: An ensemble classifier. In International conference on intelligent data communication technologies and internet of things (ICICI) 2018. Springer, 758–763.

Robert Phillipson, M. Rannut, and Tove Skutnabb-Kangas. 1994. Linguistic Human Rights: Overcoming Linguistic Discrimination. Tove Skutnabb-Kangas. [link]

J. Ross Quinlan. 1996. Learning decision tree classifiers. ACM Computing Surveys (CSUR) 28, 1 (1996), 71–72.

Tamires Martins Rezende, Sílvia Grasiella Moreira Almeida, and Frederico Gadelha Guimarães. 2021. Development and validation of a Brazilian sign language database for human gesture recognition. Computação Neural e Aplicações (2021).

Steven J Rigatti. 2017. Random forest. Journal of Insurance Medicine 47, 1 (2017), 31–39.

E.M. Rogers. 1983. DIFFUSION OF INNOVATIONS 3RD E REV. Free Press. [link]

Adriel Santos, Iago Bacurau, Jayne de Morais Silva, Talles Viana, and Robson Feitosa. 2019. Rede Neural Artificial Convolucional Aplicada ao Reconhecimento de Configuração de Mão nos Símbolos de 0 a 9 da Língua Brasileira de Sinais (LIBRAS). 21–24. DOI: 10.5753/sbsi.2019.7432

Aakanksha Sharaff and Harshil Gupta. 2019. Extra-tree classifier with metaheuristics approach for email classification. In Advances in Computer Communication and Computational Sciences: Proceedings of IC4S 2018. Springer, 189–197.

Luis Alvaro de Lima Silva, Lori RF Machado, and Leonardo Emmendorfer. 2024. A Case and Cluster-Based Framework for Reuse and Prioritization in Software Testing. In Proceedings of the 20th Brazilian Symposium on Information Systems. 1–10.

Andrey Vakunov, Chuo-Ling Chang, Fan Zhang, George Sung, Matthias Grundmann, and Valentin Bazarevsky. 2020. MediaPipe Hands: On-device Real-time Hand Tracking. [link].

Jiaheng Zhang, Zhiyong Fang, Yupeng Zhang, and Dawn Song. 2020. Zero knowledge proofs for decision tree predictions and accuracy. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security. 2039–2053.