Sign2Sign - A First Attempt
Abstract
At the core of human engagement is communication. While technological advances enable convenient translation for different language speakers to communicate, millions of Deaf, Mute, and Hard-of-Hearing people still face immense hurdles due to the lack of accessible tools to facilitate direct sign language translation. Our project aims to build a Sign2Sign direct immersive translation tool using WebXR that takes input from any accessible camera and produces output in WebXR-supported platforms. This paper presents the preliminary results of direct translation between ten gestures of American and Chinese Sign Languages for sign language translations in an immersive environment.
Keywords:
communication, sign language, translation, XR
References
Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M. G., Lee, J., Chang, W.-T., Hua, W., Georg, M., and Grundmann, M. (2019). Mediapipe: A framework for building perception pipelines.
Moryossef, A. (2023). sign.mt: Real-time multilingual sign language translation application.
Núñez-Marcos, A., de Viñaspre, O. P., and Labaka, G. (2023). A survey on sign language machine translation. Expert Systems with Applications, 213:118993.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). PyTorch: an imperative style, high-performance deep learning library. Curran Associates Inc., Red Hook, NY, USA.
Yin, K., Moryossef, A., Hochgesang, J., Goldberg, Y., and Alikhani, M. (2021). Including signed languages in natural language processing. In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7347–7360, Online. Association for Computational Linguistics.
Moryossef, A. (2023). sign.mt: Real-time multilingual sign language translation application.
Núñez-Marcos, A., de Viñaspre, O. P., and Labaka, G. (2023). A survey on sign language machine translation. Expert Systems with Applications, 213:118993.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). PyTorch: an imperative style, high-performance deep learning library. Curran Associates Inc., Red Hook, NY, USA.
Yin, K., Moryossef, A., Hochgesang, J., Goldberg, Y., and Alikhani, M. (2021). Including signed languages in natural language processing. In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7347–7360, Online. Association for Computational Linguistics.
Published
2024-09-30
How to Cite
WENG, Titus; LEE, SingChun.
Sign2Sign - A First Attempt. In: WORKSHOP OF UNDERGRADUATE WORKS - SYMPOSIUM ON VIRTUAL AND AUGMENTED REALITY (SVR), 26. , 2024, Manaus/AM.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 45-48.
DOI: https://doi.org/10.5753/svr_estendido.2024.244070.
