Towards a Realistic Libras to Portuguese Translation

• David Vinicius da Silva UFPR
• Valter Estevam IFPR
• David Menotti UFPR

Resumo

The Brazilian Sign Language (Libras) is practiced by a small portion of the population, indicating a communication gap between deaf and hearing individuals. Thus, developing tools and techniques to mitigate this problem can be of great value. This paper introduces the Sign Language Translation (SLT) problem in the Libras context and proposes a method to translate sentences from Libras to Portuguese without human annotation of signs. The translation is performed through a sequence-to-sequence architecture based on Transformers due to their successful applications in neural machine translation and image or video captioning tasks. As our sign-to-sentence approach requires pairs of video and sentences in an amount unavailable at this moment, we also introduce a new dataset based on the translation of the Bible into Libras provided by Jehovah's Witnesses. The dataset contains annotations of 1,128 videos with 612 seconds average length and three different translations of the Bible, totaling a vocabulary of 37,466 words. Our model acquires BLEU@1 and METEOR scores of 16.19% and 6.75%, respectively. Although our results are still limited, it represents a meaningful step toward developing research on this problem.