Applying Usability Testing to Conversational Systems in an Internet of Things Context: An Industry Experience Report
Resumo
The use of conversational systems has intensified in real-world applications, particularly in Internet of Things scenarios, where natural language interaction is considered a means of reducing system complexity. However, usability and user experience issues still affect the adoption and effectiveness of such systems. This paper presents an experience report on the conduct of usability tests on a conversational system applied in an Internet of Things context, aiming to analyze user experience and identify usability problems from the perspective of end users. The experience was structured using the Goal–Question–Metric framework, which supported the definition of analysis goals, guiding questions, and evaluation metrics. The usability tests involved representative tasks and the collection of quantitative and qualitative data, including task success rate, execution time, total questions per task, and users’ subjective perceptions. The results highlight recurring issues related to understanding system responses, feedback during interaction, and the adequacy of the conversational flow for the usage context. The findings provide exploratory insights into the design and evaluation of conversational systems, focusing on human and contextual aspects in human–computer interaction.Referências
Geszten, D., Hámornik, B. P., and Hercegfi, K. (2024). Team usability testing: development and validation of a groupware usability evaluation method. Cognition, Technology & Work, 26:487–506.
Hura, S. L. (2017). Usability testing of spoken conversational systems. J. Usability Studies, 12(4):155–163.
Lewis, C. (1982). Using the “thinking aloud method”. In Cognitive Interface Design, pages 1–12. IBM T.J. Watson Research Center.
Luger, E. and Sellen, A. (2016). Like having a really bad pa: The gulf between user expectation and experience of conversational agents. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 5286–5297.
McTear, M., Callejas, Z., and Griol, D. (2016). The Conversational Interface: Talking to Smart Devices. Springer, Cham, Switzerland.
Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. In CHI ’94: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 152–158, New York, NY, USA. ACM Press.
Nielsen, J. (2000). Why you only need to test with 5 users. Nielsen Norman Group Alertbox. Accessed: 2026-01-31.
Nielsen, J. (2012). Usability 101: Introduction to usability. Nielsen Norman Group. Accessed: 2026-01-31.
Nielsen, J. and Molich, R. (1990). Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 249–256. ACM.
Pressman, R. S. and Maxim, B. R. (2020). Software Engineering: A Practitioner’s Approach. McGraw-Hill, New York, USA, 9 edition.
Rubin, J. and Chisnell, D. (2008). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley Publishing, Inc., Indianapolis, IN, 2nd edition.
Scherer, K. R. (2005). What are emotions? and how can they be measured? Social Science Information, 44(4):693–727.
Scherer, K. R., Shuman, V., Fontaine, J. R. J., and Soriano, C. (2013). The grid meets the wheel: Assessing emotional feeling via self-report. In Fontaine, J. R. J., Scherer, K. R., and Soriano, C., editors, Components of Emotional Meaning: A sourcebook, pages 281–298. Oxford University Press, Oxford.
Wazlawick, R. S. (2014). Metodologia de Pesquisa para Ciência da Computação. Elsevier, Rio de Janeiro, 2 edition.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2024). Experimentation in Software Engineering. Springer, 2 edition.
Zhang, Y., Zaidi, M., et al. (2022). Machine-learned and human brain representations of speech-in-noise are mutually informative. Frontiers in Computational Neuroscience, 16:1057439.
Hura, S. L. (2017). Usability testing of spoken conversational systems. J. Usability Studies, 12(4):155–163.
Lewis, C. (1982). Using the “thinking aloud method”. In Cognitive Interface Design, pages 1–12. IBM T.J. Watson Research Center.
Luger, E. and Sellen, A. (2016). Like having a really bad pa: The gulf between user expectation and experience of conversational agents. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 5286–5297.
McTear, M., Callejas, Z., and Griol, D. (2016). The Conversational Interface: Talking to Smart Devices. Springer, Cham, Switzerland.
Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. In CHI ’94: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 152–158, New York, NY, USA. ACM Press.
Nielsen, J. (2000). Why you only need to test with 5 users. Nielsen Norman Group Alertbox. Accessed: 2026-01-31.
Nielsen, J. (2012). Usability 101: Introduction to usability. Nielsen Norman Group. Accessed: 2026-01-31.
Nielsen, J. and Molich, R. (1990). Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 249–256. ACM.
Pressman, R. S. and Maxim, B. R. (2020). Software Engineering: A Practitioner’s Approach. McGraw-Hill, New York, USA, 9 edition.
Rubin, J. and Chisnell, D. (2008). Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley Publishing, Inc., Indianapolis, IN, 2nd edition.
Scherer, K. R. (2005). What are emotions? and how can they be measured? Social Science Information, 44(4):693–727.
Scherer, K. R., Shuman, V., Fontaine, J. R. J., and Soriano, C. (2013). The grid meets the wheel: Assessing emotional feeling via self-report. In Fontaine, J. R. J., Scherer, K. R., and Soriano, C., editors, Components of Emotional Meaning: A sourcebook, pages 281–298. Oxford University Press, Oxford.
Wazlawick, R. S. (2014). Metodologia de Pesquisa para Ciência da Computação. Elsevier, Rio de Janeiro, 2 edition.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2024). Experimentation in Software Engineering. Springer, 2 edition.
Zhang, Y., Zaidi, M., et al. (2022). Machine-learned and human brain representations of speech-in-noise are mutually informative. Frontiers in Computational Neuroscience, 16:1057439.
Publicado
19/07/2026
Como Citar
GOMES, Camilla Rosas; SANTOS, Marcos William Cardoso dos; MANZONI, Felipe S.; MARQUES, Leonardo; EKWOGE, Oswald Mesumbe.
Applying Usability Testing to Conversational Systems in an Internet of Things Context: An Industry Experience Report. In: WORKSHOP SOBRE ASPECTOS SOCIAIS, HUMANOS E ECONÔMICOS DE SOFTWARE (WASHES), 11. , 2026, Gramado/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 25-35.
ISSN 2763-874X.
DOI: https://doi.org/10.5753/washes.2026.23513.
