LibrasDetec: um Componente de Detecção de Movimentos para Karaokê em Libras
Resumo
The entertainment market has been expanding rapidly, driven in part by increased internet access, which has also contributed to the growth of the digital games industry. However, despite this expansion, investments in accessibility remain limited, compromising the quality of games for people with disabilities. In the context of entertainment or serious games, particularly regarding accessibility for deaf users, motion detection emerges as a promising strategy to enable interaction through gestures in their native language. In this scenario, we propose LibrasDetec, a component for detecting and evaluating Brazilian Sign Language (Libras) gestures, integrated into the educational game Libraskê. The solution captures user gestures via webcam and compares them with reference videos, providing real-time scoring. To validate the proposed component, we conducted tests with 12 participants from three different user profiles: deaf individuals, interpreters, and non-signers. The results showed a satisfactory alignment between automated and human evaluations, especially among deaf users, suggesting that LibrasDetec is a viable approach to enhance accessibility and engagement in serious games focused on Libras.
Palavras-chave:
serious game, motion detection, accessible game, sign language
Referências
H. Akaike. 1974. A new look at the statistical model identification. IEEE Trans. Automat. Control 19, 6 (1974), 716–723. DOI: 10.1109/TAC.1974.1100705
Diego Roberto Antunes and Janaine Daiane Rodrigues. 2021. Endless Running Game to Support Sign Language Learning by Deaf Children. In International Conference on Human-Computer Interaction. Springer International Publishing, Cham.
G. E. P. Box and D. R. Cox. 2018. An Analysis of Transformations. Journal of the Royal Statistical Society: Series B (Methodological) 26, 2 (12 2018), 211–243. DOI: 10.1111/j.2517-6161.1964.tb00553.x
Brasil. 2002. Lei nº 10.436, de 24 de abril de 2002. [link] Acesso em: 17 jul. 2025.
Alexander Brettmann, Jakob Grävinghoff, Marlene Rüschoff, and MarieWesthues. 2025. Breaking the Barriers: Video Vision Transformers for Word-Level Sign Language Recognition. arXiv preprint arXiv:2504.07792 (2025). DOI: 10.48550/arXiv.2504.07792
Z. Cao, G. Hidalgo, T. Simon, S. Wei, and Y. Sheikh. 2021. OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence 43 (2021), 172–186.
John M Chambers. 2018. Graphical methods for data analysis. Chapman and Hall/CRC.
William S. Cleveland. 1979. Robust Locally Weighted Regression and Smoothing Scatterplots. J. Amer. Statist. Assoc. 74, 368 (1979), 829–836. DOI: 10.1080/01621459.1979.10481038
R Dennis Cook. 1977. Detection of influential observation in linear regression. Technometrics 19, 1 (1977), 15–18.
M. Cullinan and L. L. Wood. 2024. Getting Inspired: A Qualitative Study on the Use of the Inspirisles Role-Playing Game to Teach Middle Schoolers American Sign Language. Simulation & Gaming 55, 2 (2024), 267–280. DOI: 10.1177/10468781241229614 Original work published 2024.
Ralph D’Agostino and E. S. Pearson. 1973. Tests for departure from normality. Empirical results for the distributions of b2 and b1. Biometrika 60, 3 (12 1973), 613–622. DOI: 10.1093/biomet/60.3.613
P. T. Dinh, T. T. Nguyen, H. Q. Le, V. H. Nguyen, T. P. Nguyen, H. N. Tran, and N. T. Nguyen. 2025. Sign Language Recognition: A Large-Scale Multi-View Dataset and Comprehensive Evaluation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). [link]
Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. 2017. RMPE: Regional Multi-person Pose Estimation. In 2017 IEEE International Conference on Computer Vision (ICCV). 2353–2362. DOI: 10.1109/ICCV.2017.256
R. A. Fisher. 1992. Statistical Methods for Research Workers. Springer New York, New York, NY, 66–70. DOI: 10.1007/978-1-4612-4380-9_6
Manolis Fragkiadakis and Peter van der Putten. 2021. Sign and Search: Sign Search Functionality for Sign Language Lexica. In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL). Association for Machine Translation in the Americas, Virtual, 23–32. [link]
A. Halder, A. ans Tayade. 2021. Real-time vernacular sign language recognition using mediapipe and machine learning. 2 (2021), 9–17.
Valerie Henderson, Seungyon Lee, Helene Brashear, Harley Hamilton, Thad Starner, and Steven Hamilton. 2005. Development of an American Sign Language game for deaf children. Association for Computing Machinery, 70–79. DOI: 10.1145/1109540.1109550
Carlos M. Jarque and Anil K. Bera. 1987. A Test for Normality of Observations and Regression Residuals. International Statistical Review / Revue Internationale de Statistique 55, 2 (1987), 163–172. [link]
Z. Li, X. Wang, Y. Wang, and J. Zhang. 2024. YOLOv5-DTW: Gesture recognition based on YOLOv5 and dynamic time warping for digital media design. Journal of Applied Science and Engineering 29, 2 (2024), 1–10. [link]
C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C. Chang, M. Yong, J. Lee, W. Chang, W. Hua, M. Georg, and M. Grundmann. 2019. MediaPipe: A Framework for Perceiving and Processing Reality. In Third Workshop on Computer Vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR) 2019.
C. R. Ramos. 2022. A Língua de Sinais dos Surdos Brasileiros. Arara Azul, Petrópolis-SP.
J. B. Ramsey. 1969. Tests for Specification Errors in Classical Linear Least-Squares Regression Analysis. Journal of the Royal Statistical Society: Series B (Methodological) 31, 2 (1969), 350–371. DOI: 10.1111/j.2517-6161.1969.tb00796.x
P. Senin. 2008. Dynamic Time Warping Algorithm Review. 855 (2008), 23–40.
A. P. Shanker and A.N. Rajagopalan. 2007. Off-line signature verification using DTW. 28 (2007), 1407–1414.
Luca Ulrich et al. 2024. SIGNIFY: Leveraging Machine Learning and Gesture Recognition for Sign Language Teaching through a Serious Game. Future Internet 16, 12 (2024), 447. DOI: 10.3390/fi16120447
Y. Wang and X. Liu. 2019. DTWNet: A Dynamic Time Warping Network. In Advances in Neural Information Processing Systems, Vol. 32. [link]
Diego Roberto Antunes and Janaine Daiane Rodrigues. 2021. Endless Running Game to Support Sign Language Learning by Deaf Children. In International Conference on Human-Computer Interaction. Springer International Publishing, Cham.
G. E. P. Box and D. R. Cox. 2018. An Analysis of Transformations. Journal of the Royal Statistical Society: Series B (Methodological) 26, 2 (12 2018), 211–243. DOI: 10.1111/j.2517-6161.1964.tb00553.x
Brasil. 2002. Lei nº 10.436, de 24 de abril de 2002. [link] Acesso em: 17 jul. 2025.
Alexander Brettmann, Jakob Grävinghoff, Marlene Rüschoff, and MarieWesthues. 2025. Breaking the Barriers: Video Vision Transformers for Word-Level Sign Language Recognition. arXiv preprint arXiv:2504.07792 (2025). DOI: 10.48550/arXiv.2504.07792
Z. Cao, G. Hidalgo, T. Simon, S. Wei, and Y. Sheikh. 2021. OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence 43 (2021), 172–186.
John M Chambers. 2018. Graphical methods for data analysis. Chapman and Hall/CRC.
William S. Cleveland. 1979. Robust Locally Weighted Regression and Smoothing Scatterplots. J. Amer. Statist. Assoc. 74, 368 (1979), 829–836. DOI: 10.1080/01621459.1979.10481038
R Dennis Cook. 1977. Detection of influential observation in linear regression. Technometrics 19, 1 (1977), 15–18.
M. Cullinan and L. L. Wood. 2024. Getting Inspired: A Qualitative Study on the Use of the Inspirisles Role-Playing Game to Teach Middle Schoolers American Sign Language. Simulation & Gaming 55, 2 (2024), 267–280. DOI: 10.1177/10468781241229614 Original work published 2024.
Ralph D’Agostino and E. S. Pearson. 1973. Tests for departure from normality. Empirical results for the distributions of b2 and b1. Biometrika 60, 3 (12 1973), 613–622. DOI: 10.1093/biomet/60.3.613
P. T. Dinh, T. T. Nguyen, H. Q. Le, V. H. Nguyen, T. P. Nguyen, H. N. Tran, and N. T. Nguyen. 2025. Sign Language Recognition: A Large-Scale Multi-View Dataset and Comprehensive Evaluation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). [link]
Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. 2017. RMPE: Regional Multi-person Pose Estimation. In 2017 IEEE International Conference on Computer Vision (ICCV). 2353–2362. DOI: 10.1109/ICCV.2017.256
R. A. Fisher. 1992. Statistical Methods for Research Workers. Springer New York, New York, NY, 66–70. DOI: 10.1007/978-1-4612-4380-9_6
Manolis Fragkiadakis and Peter van der Putten. 2021. Sign and Search: Sign Search Functionality for Sign Language Lexica. In Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL). Association for Machine Translation in the Americas, Virtual, 23–32. [link]
A. Halder, A. ans Tayade. 2021. Real-time vernacular sign language recognition using mediapipe and machine learning. 2 (2021), 9–17.
Valerie Henderson, Seungyon Lee, Helene Brashear, Harley Hamilton, Thad Starner, and Steven Hamilton. 2005. Development of an American Sign Language game for deaf children. Association for Computing Machinery, 70–79. DOI: 10.1145/1109540.1109550
Carlos M. Jarque and Anil K. Bera. 1987. A Test for Normality of Observations and Regression Residuals. International Statistical Review / Revue Internationale de Statistique 55, 2 (1987), 163–172. [link]
Z. Li, X. Wang, Y. Wang, and J. Zhang. 2024. YOLOv5-DTW: Gesture recognition based on YOLOv5 and dynamic time warping for digital media design. Journal of Applied Science and Engineering 29, 2 (2024), 1–10. [link]
C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C. Chang, M. Yong, J. Lee, W. Chang, W. Hua, M. Georg, and M. Grundmann. 2019. MediaPipe: A Framework for Perceiving and Processing Reality. In Third Workshop on Computer Vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR) 2019.
C. R. Ramos. 2022. A Língua de Sinais dos Surdos Brasileiros. Arara Azul, Petrópolis-SP.
J. B. Ramsey. 1969. Tests for Specification Errors in Classical Linear Least-Squares Regression Analysis. Journal of the Royal Statistical Society: Series B (Methodological) 31, 2 (1969), 350–371. DOI: 10.1111/j.2517-6161.1969.tb00796.x
P. Senin. 2008. Dynamic Time Warping Algorithm Review. 855 (2008), 23–40.
A. P. Shanker and A.N. Rajagopalan. 2007. Off-line signature verification using DTW. 28 (2007), 1407–1414.
Luca Ulrich et al. 2024. SIGNIFY: Leveraging Machine Learning and Gesture Recognition for Sign Language Teaching through a Serious Game. Future Internet 16, 12 (2024), 447. DOI: 10.3390/fi16120447
Y. Wang and X. Liu. 2019. DTWNet: A Dynamic Time Warping Network. In Advances in Neural Information Processing Systems, Vol. 32. [link]
Publicado
10/11/2025
Como Citar
LIMA, Manuella Aschoff; FRANÇA, Daniel de; SILVA NETO, Arnor da; SOUZA, Daniel Faustino de; OMAIA, Derzu; ARAÚJO, Tiago Maritan de.
LibrasDetec: um Componente de Detecção de Movimentos para Karaokê em Libras. In: BRAZILIAN SYMPOSIUM ON MULTIMEDIA AND THE WEB (WEBMEDIA), 31. , 2025, Rio de Janeiro/RJ.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 340-348.
DOI: https://doi.org/10.5753/webmedia.2025.15182.
