Human-ChatBot Interaction: measuring the psychophysiological reactions of chatbot users

  • Helen Santos Picoli USP
  • Cynthya Letícia Teles De Oliveira USP
  • Lucas Padilha M. De Oliveira USP
  • Marcelo G. Manzato USP
  • Kamila Rios Da Hora Rodrigues USP


Affective computing and its applications have gained popularity in recent years. However, recognizing emotions is a challenge. In Computing, the artifacts used to recognize emotions are instruments of self-report or data collection through biological sensors. Interaction through chatbots, in turn, has grown and brings with it challenges related to how effective the system is and how satisfied the user is when using it. We carried out an experiment with the aim of learning more about the characteristics of human-non-human interaction. As a result, the expressive responses of users and the chatbots with whom they interacted are the focus of this human-chatbot interaction paper. We gathered face registration, body movements, self-report, and peripheral signals such as the electrocardiogram (ECG), valence, and arousal estimations of the various emotional changes during the engagement period with the Bob chatbot, a conversational recommendation system, that operates through the WhatsApp app, and recommends restaurants to the user based on information such as location. The results were encouraging; users felt comfortable and were receptive to a new conversational tool and the ECG sensors attached to the thorax, which motivates us to make improvements for future tests.

Palavras-chave: Emotional responses, chat-bot, sensors, physiological data, self-report data


Foteini Agrafioti, Dimitris Hatzinakos, and Adam K. Anderson. 2012. ECG Pattern Analysis for Emotion Detection. IEEE Transactions on Affective Computing 3, 1 (2012), 102–115.

Amani Albraikan, Basim Hafidh, and Abdulmotaleb El Saddik. 2018. iAware: A real-time emotional biofeedback system based on physiological signals. IEEE Access 6 (2018), 78780–78789.

R S Bexton, H O Vallin, and A J Camm. 1986. Diurnal variation of the QT interval–influence of the autonomic nervous system. Heart 55, 3 (1986), 253–258. arXiv:

Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1 (1994), 49–59.

Paul Ekman, Wallace V. Freisen, and Sonia Ancoli. 1980. Facial signs of emotional experience. Journal of Personality and Social Psychology 39 (1980), 1125–1134.

Elaine CS Hayashi, Julián E Gutiérrez Posada, Vanessa RML Maike, and M Cecília C Baranauskas. 2016. Exploring new formats of the Self-Assessment Manikin in the design with children. In Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems. ACM, São Paulo, Brazil, 1–10.

Paris Hsu, Jingshu Zhao, Kehan Liao, Tianyi Liu, and Chen Wang. 2017. AllergyBot: A Chatbot technology intervention for young adults with food allergies dining out. In Proceedings of the 2017 CHI conference extended abstracts on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 74–79.

Lorenz Cuno Klopfenstein, Saverio Delpriori, Silvia Malatini, and Alessandro Bogliolo. 2017. The rise of bots: A survey of conversational interfaces, patterns, and paradigms. In Proceedings of the 2017 conference on designing interactive systems. New York, NY, 555–565.

Rollin McCraty. 2015. Science of the heart: Exploring the role of the heart in human performance. HeartMath Research Center, Institute of HeartMath, CA.

Sintija Petrovica, Alla Anohina-Naumeca, and Hazım Kemal Ekenel. 2017. Emotion recognition in affective tutoring systems: Collection of ground-truth data. Procedia Computer Science 104 (2017), 437–444.

Kanlaya Rattanyu and Makoto Mizukawa. 2011. Emotion Recognition Based on ECG Signals for Service Robots in the Intelligent Space During Daily Life. Journal of Advanced Computational Intelligence and Intelligent Informatics 15, 5 (2011), 582–591.

Byron Reeves and Clifford Nass. 1996. The Media Equation: How People Treat Computers, Television, and New Media like Real People and Places. Cambridge University Press, USA.

Luca Romeo, Andrea Cavallo, Lucia Pepa, Nadia Berthouze, and Massimiliano Pontil. 2019. Multiple instance learning for emotion recognition using physiological signals. IEEE Transactions on Affective Computing 13, 1 (2019), 389 – 407.

Jim Torresen. 2018. A review of future and ethical perspectives of robotics and AI. Frontiers in Robotics and AI 4 (2018), 75.

Rafael Wampfler, Severin Klingler, Barbara Solenthaler, Victor Schinazi, and Markus Gross. 2019. Affective state prediction in a mobile setting using wearable biometric sensors and stylus. In Proceedings of the 12th International Conference on Educational Data Mining, EDM 2019, Montréal, Canada, July 2-5, 2019. International Educational Data Mining Society (IEDMS) 2019. Université du Québec; Polytechnique Montréal, International Educational Data Mining Society (IEDMS), Montréal, Canada, 198–207.

Tianhua Xie, Mingliang Cao, and Zhigeng Pan. 2020. Applying self-assessment manikin (sam) to evaluate the affective arousal effects of vr games. In Proceedings of the 2020 3rd International Conference on Image and Graphics Processing. ACM, Singapore, 134–138.

Bobo Zhao, Zhu Wang, Zhiwen Yu, and Bin Guo. 2018. EmotionSense: Emotion recognition based on wearable wristband. In 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI). IEEE, IEEE, Guangzhou, China, 346–355.
PICOLI, Helen Santos; OLIVEIRA, Cynthya Letícia Teles De; OLIVEIRA, Lucas Padilha M. De; MANZATO, Marcelo G.; RODRIGUES, Kamila Rios Da Hora. Human-ChatBot Interaction: measuring the psychophysiological reactions of chatbot users. In: SIMPÓSIO BRASILEIRO DE SISTEMAS MULTIMÍDIA E WEB (WEBMEDIA), 28. , 2022, Curitiba. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2022 . p. 405-408.