Leveraging Sign Language Processing with Formal SignWriting and Deep Learning Architectures

• Fernando de Almeida Freitas USP https://orcid.org/0009-0008-4937-9097
• Sarajane Marques Peres USP https://orcid.org/0000-0003-3551-6480
• Otávio de Paula Albuquerque USP https://orcid.org/0000-0002-2504-3707
• Marcelo Fantinato USP https://orcid.org/0000-0001-6261-1497

Resumo

Advances in sign language processing have not adequately kept pace with the tremendous progress that has been made in oral language processing. This fact serves as motivation for conducting research on the potential utilization of deep learning models within the domain of sign language processing. In this paper, we present a method that utilizes deep learning to build a latent and generalizable representation space for signs, leveraging Formal SignWriting notation and the concept of sentence-based representation to effectively address sign language tasks, such as sign classification. Extensive experiments demonstrate the potential of this method, achieving an average accuracy of 81% on a subset of 70 signs with only 889 training data and 69% on a subset of 338 signs with 3, 871 training data.