RoboSign: Improving Interaction Between Service-Robot and Hearing Impairments People

• Aron Caiuá V. De Brito UNEB
• Ana Patrícia F. M. Mascarenhas UNEB
• Marco A.C. Simões UNEB
• Ivanoé J. Rodowanski UNEB
• Jorge Alberto P. De Campos UNEB
• Josemar Rodrigues De Souza UNEB
• Jose Grimaldo Da Silva Filho UNEB

Resumo

Service robots assist humans in routine activities such as domestic tasks. Their interaction with humans, e.g., during communication, is commonly done through voice recognition functionalities. However, voice interaction limits robot usage, making it inaccessible to hearing-impaired people. This paper presents RoboSign, an interaction solution between a service robot and people with hearing impairments through command recognition in Brazilian Sign language (Libras). RoboSign includes a simplified dataset from scratch for a computer with low computational power, training a recurrent neural network model using MediaPipe Landmark Detector and Long ShortTerm Memory (LSTM) to perform signal recognition. The model was validated during training with unknown data, reaching 99% average accuracy, and evaluated in real-time using a camera, reaching 85% average accuracy. These results demonstrate that it is possible to perform human-robot interaction through Libras recognition with considerable accuracy without high computational cost due to simplified data collection and the LSTM algorithm for network training.

Palavras-chave: Training, Sign language, Accuracy, Recurrent neural networks, Service robots, Computational modeling, Auditory system, Speech recognition, Real-time systems, Long short term memory