Integration of Transformer-Based Language Models with Humanoid Robots: A Case Study Using NAO and TinyLlama for Real-Time Conversational Interaction
Resumo
This work presents a spoken dialogue system integrating the NAO humanoid robot with the TinyLlama-1.1B-Chat-v1.0 language model through a modular pipeline architecture that facilitates maintenance and integration with commercial robotic platforms, combining NAO’s native ASR and TTS modules with the language model; experimental results indicate a WER of 15% for a 163-word vocabulary, an average response latency of 4.2 seconds, and the ability to generate contextually appropriate responses.
Referências
T. Brown et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 1877–1901, 2020.
A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, pp. 5998–6008, 2017.
H. S. Ahn et al., “Hospital receptionist robot v2: Design for enhancing verbal interaction with social skills,” in IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2019, pp. 1–7.
G. Skantze and B. Irfan, “Applying general turn-taking models to conversational human-robot interaction,” in Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2025, pp. 859–868.
A. Défossez et al., “Moshi: A speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024.
X. Wang et al., “Freeze-Omni: A smart and low latency speech-to-speech dialogue model with frozen LLM,” in International Conference on Machine Learning (ICML), 2025.
E. Pot et al., “Choregraphe: A graphical tool for humanoid robot programming,” in IEEE International Symposium on Robot and Human Interactive Communication, 2009, pp. 46–51.
P. Zhang et al., “TinyLlama: An open-source small language model,” arXiv preprint arXiv:2401.02385, 2024.
G. Skantze, “Turn-taking in conversational systems and human-robot interaction: A review,” Computer Speech & Language, vol. 67, 101178, 2021.
T. Fong, I. Nourbakhsh, and K. Dautenhahn, “A survey of socially interactive robots,” Robotics and Autonomous Systems, vol. 42, no. 3–4, pp. 143–166, 2003.
B. R. Duffy, “Anthropomorphism and the social robot,” Robotics and Autonomous Systems, vol. 42, no. 3–4, pp. 177–190, 2003.
M. Heerink et al., “Assessing acceptance of assistive social agent technology by older adults: The Almere model,” International Journal of Social Robotics, vol. 2, no. 4, pp. 361–375, 2010.
S. Baron-Cohen, A. M. Leslie, and U. Frith, “Does the autistic child have a ‘theory of mind’? A case study,” Cognition, vol. 21, no. 1, pp. 37–46, 1985.
B. Scassellati, “Theory of mind for a humanoid robot,” Autonomous Robots, vol. 12, no. 1, pp. 13–24, 2002.
A. Radford et al., “Language models are unsupervised multitask learners,” OpenAI Blog, 2019.
G. Hinton et al., “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, 2012.
D. Yu and L. Deng, Automatic Speech Recognition: A Deep Learning Approach. London, UK: Springer, 2016.
P. Taylor, Text-to-Speech Synthesis. Cambridge, UK: Cambridge University Press, 2009.
A. van den Oord et al., “WaveNet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016.
D. Gouaillier et al., “Mechatronic design of NAO humanoid,” in IEEE International Conference on Robotics and Automation (ICRA), 2009, pp. 769–774.
Aldebaran Robotics, NAO Technical Documentation. SoftBank Robotics, 2021. Available: [link]. Accessed: Sep. 2, 2025.
G. Ficht et al., “NimbRo-OP2X: Adult-sized open-source 3D printed humanoid robot,” in IEEE-RAS International Conference on Humanoid Robots (Humanoids), 2018, pp. 1–9.
T. Belpaeme et al., “Social robots for education: A review,” Science Robotics, vol. 3, no. 21, eaat5954, 2018.
Y. Goldberg, “A primer on neural network models for natural language processing,” Journal of Artificial Intelligence Research, vol. 57, pp. 345–420, 2016.
D. Han et al., “Rethinking channel dimensions for efficient model design,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 732–741.
J. Nielsen, Usability Engineering. San Francisco, CA, USA: Morgan Kaufmann, 1994.
A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, pp. 5998–6008, 2017.
H. S. Ahn et al., “Hospital receptionist robot v2: Design for enhancing verbal interaction with social skills,” in IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2019, pp. 1–7.
G. Skantze and B. Irfan, “Applying general turn-taking models to conversational human-robot interaction,” in Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2025, pp. 859–868.
A. Défossez et al., “Moshi: A speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024.
X. Wang et al., “Freeze-Omni: A smart and low latency speech-to-speech dialogue model with frozen LLM,” in International Conference on Machine Learning (ICML), 2025.
E. Pot et al., “Choregraphe: A graphical tool for humanoid robot programming,” in IEEE International Symposium on Robot and Human Interactive Communication, 2009, pp. 46–51.
P. Zhang et al., “TinyLlama: An open-source small language model,” arXiv preprint arXiv:2401.02385, 2024.
G. Skantze, “Turn-taking in conversational systems and human-robot interaction: A review,” Computer Speech & Language, vol. 67, 101178, 2021.
T. Fong, I. Nourbakhsh, and K. Dautenhahn, “A survey of socially interactive robots,” Robotics and Autonomous Systems, vol. 42, no. 3–4, pp. 143–166, 2003.
B. R. Duffy, “Anthropomorphism and the social robot,” Robotics and Autonomous Systems, vol. 42, no. 3–4, pp. 177–190, 2003.
M. Heerink et al., “Assessing acceptance of assistive social agent technology by older adults: The Almere model,” International Journal of Social Robotics, vol. 2, no. 4, pp. 361–375, 2010.
S. Baron-Cohen, A. M. Leslie, and U. Frith, “Does the autistic child have a ‘theory of mind’? A case study,” Cognition, vol. 21, no. 1, pp. 37–46, 1985.
B. Scassellati, “Theory of mind for a humanoid robot,” Autonomous Robots, vol. 12, no. 1, pp. 13–24, 2002.
A. Radford et al., “Language models are unsupervised multitask learners,” OpenAI Blog, 2019.
G. Hinton et al., “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, 2012.
D. Yu and L. Deng, Automatic Speech Recognition: A Deep Learning Approach. London, UK: Springer, 2016.
P. Taylor, Text-to-Speech Synthesis. Cambridge, UK: Cambridge University Press, 2009.
A. van den Oord et al., “WaveNet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016.
D. Gouaillier et al., “Mechatronic design of NAO humanoid,” in IEEE International Conference on Robotics and Automation (ICRA), 2009, pp. 769–774.
Aldebaran Robotics, NAO Technical Documentation. SoftBank Robotics, 2021. Available: [link]. Accessed: Sep. 2, 2025.
G. Ficht et al., “NimbRo-OP2X: Adult-sized open-source 3D printed humanoid robot,” in IEEE-RAS International Conference on Humanoid Robots (Humanoids), 2018, pp. 1–9.
T. Belpaeme et al., “Social robots for education: A review,” Science Robotics, vol. 3, no. 21, eaat5954, 2018.
Y. Goldberg, “A primer on neural network models for natural language processing,” Journal of Artificial Intelligence Research, vol. 57, pp. 345–420, 2016.
D. Han et al., “Rethinking channel dimensions for efficient model design,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 732–741.
J. Nielsen, Usability Engineering. San Francisco, CA, USA: Morgan Kaufmann, 1994.
Publicado
19/07/2026
Como Citar
SOUZA, Vitor Amadeu.
Integration of Transformer-Based Language Models with Humanoid Robots: A Case Study Using NAO and TinyLlama for Real-Time Conversational Interaction. In: SEMINÁRIO INTEGRADO DE SOFTWARE E HARDWARE (SEMISH), 53. , 2026, Gramado/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 962-973.
ISSN 2595-6205.
DOI: https://doi.org/10.5753/semish.2026.21089.
