Reinforcement Learning for Shoulder Control of the NAO Robot Using Q-Learning: Tests in Real and Simulated Environments
Resumo
This paper presents a Q-Learning approach for controlling the NAO humanoid robot’s ShoulderPitch joint, motivated by the limitations of traditional PID controllers in handling nonlinearities and inter-joint coupling. The methodology involves training in CoppeliaSim simulation before deployment on the physical robot, with a reward function based on absolute angular error relative to a 45° target and discretized state space. Results demonstrate policy convergence in simulation and real-world implementation, with the performance gap providing insights into sim-to-real transfer challenges and establishing a foundation for future multi-joint applications.
Referências
J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” The International Journal of Robotics Research, vol. 32, no. 11, pp. 1238–1274, 2013.
D. Gouaillier et al., “Mechatronic design of NAO humanoid,” in IEEE International Conference on Robotics and Automation (ICRA), 2009, pp. 769–774. DOI: 10.1109/ROBOT.2009.5152516.
S. Shamsuddin et al., “Humanoid robot NAO: Review of control and motion exploration,” in IEEE International Conference on Control System, Computing and Engineering, 2011, pp. 511–516.
M. González-Fierro et al., “Full-body postural control of a humanoid robot with both imitation learning and skill innovation,” International Journal of Humanoid Robotics, vol. 11, no. 2, p. 1450012, 2014.
R. Tedrake, T. W. Zhang, and H. S. Seung, “Stochastic policy gradient reinforcement learning on a simple 3D biped,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2004, vol. 3, pp. 2849–2854.
S. Kuindersma et al., “Optimization-based locomotion planning, estimation, and control design for Atlas,” Autonomous Robots, vol. 40, no. 3, pp. 429–455, 2016.
S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” Journal of Machine Learning Research, vol. 17, no. 1, pp. 1334–1373, 2016.
V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
C. J. C. H. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, no. 3–4, pp. 279–292, 1992.
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 2018.
Aldebaran, “Electronics architecture.” [Online]. Available: [link].
SoftBank Robotics, NAO Technical Documentation, 2014. [Online]. Available: [link].
E. Pot, J. Monceaux, R. Gelin, and B. Maisonnier, “Choregraphe: A graphical tool for humanoid robot programming,” in IEEE International Symposium on Robot and Human Interactive Communication, 2009, pp. 46–51.
E. Rohmer, S. P. N. Singh, and M. Freese, “V-REP: A versatile and scalable robot simulation framework,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2013, pp. 1321–1326.
Souza and Sá, “Source code.” [Online]. Available: [link].
Souza and Sá, “Videos.” [Online]. Available: [link].
D. Gouaillier et al., “Mechatronic design of NAO humanoid,” in IEEE International Conference on Robotics and Automation (ICRA), 2009, pp. 769–774. DOI: 10.1109/ROBOT.2009.5152516.
S. Shamsuddin et al., “Humanoid robot NAO: Review of control and motion exploration,” in IEEE International Conference on Control System, Computing and Engineering, 2011, pp. 511–516.
M. González-Fierro et al., “Full-body postural control of a humanoid robot with both imitation learning and skill innovation,” International Journal of Humanoid Robotics, vol. 11, no. 2, p. 1450012, 2014.
R. Tedrake, T. W. Zhang, and H. S. Seung, “Stochastic policy gradient reinforcement learning on a simple 3D biped,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2004, vol. 3, pp. 2849–2854.
S. Kuindersma et al., “Optimization-based locomotion planning, estimation, and control design for Atlas,” Autonomous Robots, vol. 40, no. 3, pp. 429–455, 2016.
S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” Journal of Machine Learning Research, vol. 17, no. 1, pp. 1334–1373, 2016.
V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, 2015.
C. J. C. H. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, no. 3–4, pp. 279–292, 1992.
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT Press, 2018.
Aldebaran, “Electronics architecture.” [Online]. Available: [link].
SoftBank Robotics, NAO Technical Documentation, 2014. [Online]. Available: [link].
E. Pot, J. Monceaux, R. Gelin, and B. Maisonnier, “Choregraphe: A graphical tool for humanoid robot programming,” in IEEE International Symposium on Robot and Human Interactive Communication, 2009, pp. 46–51.
E. Rohmer, S. P. N. Singh, and M. Freese, “V-REP: A versatile and scalable robot simulation framework,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2013, pp. 1321–1326.
Souza and Sá, “Source code.” [Online]. Available: [link].
Souza and Sá, “Videos.” [Online]. Available: [link].
Publicado
19/07/2026
Como Citar
SOUZA, Vitor Amadeu; SÁ, Hebert Azevedo.
Reinforcement Learning for Shoulder Control of the NAO Robot Using Q-Learning: Tests in Real and Simulated Environments. In: SEMINÁRIO INTEGRADO DE SOFTWARE E HARDWARE (SEMISH), 53. , 2026, Gramado/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 1004-1015.
ISSN 2595-6205.
DOI: https://doi.org/10.5753/semish.2026.21100.
