UX evaluation of IoT-based applications for Smart Cities: a rapid systematic review
Resumo
The development of systems based on the Internet of Things (IoT) technologies is becoming increasingly popular in the context of Smart Cities. Recent research in Software Engineering (SE) has investigated the characteristics of these systems and the most appropriate approaches to their design and development. IoT-based applications are strongly characterized by the interaction among multiple devices, users, and services. These characteristics make these applications complex and difficult to evaluate, particularly in terms of User Experience (UX) design. In this study, we performed a rapid systematic review to examine the methods and practices commonly employed for evaluating the UX of IoT-based applications for Smart Cities. We analyzed 43 studies covering different types of IoT-based applications, most in the area of Smart Home. Our findings indicate that user testing questionnaire-based (e.g., SUS - System Usability Scale and UEQ - User Experience Questionnaire) and interviews are the most used methods to evaluate IoT applications, while few studies mentioned user testing based on automated measurement or expert-based reviews. This work contributes to SE research by revealing the need to explore UX evaluation methods based on usage data and the combination of methods for continuous UX evaluation in the context of IoT and Smart Cities.
Referências
Abdulsattar, N. F., Abbas, A. H., Mutar, M. H., Hassan, M. H., Jubair, M. A., and Habelalmateen, M. I. (2022). An investigation study for technologies, challenges and practices of iot in smart cities. In 2022 5th International Conference on Engineering Technology and its Applications (IICETA), pages 554–557. IEEE.
Aceves Gutierrez, L. C., Martin Gutierrez, J., and Del-Rio-Guerra, M. S. (2019). Having a smarter city through digital urban interfaces: an evaluation method. Applied Sciences, 9(17):3498.
Almeida, R. L., Darin, T. G., Andrade, R. M., and de Araújo, I. L. (2018). Towards developing a practical tool to assist ux evaluation in the iot scenario. In Anais Estendidos do XXIV Simpósio Brasileiro de Sistemas Multimídia e Web, pages 91–95. SBC.
Andrade, R., Carvalho, R. M., Araújo, I. L. d., Oliveira, K. M., and Maia, M. E. (2017). What changes from ubiquitous computing to internet of things in interaction evaluation? In Int. Conf. on Distributed, Ambient, and Pervasive Interactions, pages 3–21.
Angeloni, M. P. C., Oliveira, K. M. d., Grislin-Le Strugeon, E., Duque, R., and Tirnauca, C. (2023). A review on quality in use evaluation of smart environment applications: What’s next? In Companion Proceedings of the 2023 ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pages 9–15.
Brooke, J. (2013). Sus: a retrospective. j. usability stud. 8 (2), 29–40.
Cartaxo, B., Pinto, G., and Soares, S. (2018). The role of rapid reviews in supporting decision-making in software engineering practice. In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018, pages 24–34.
Choma, J., Machado, L., de Souza, C. R., Sharp, H., Barroca, L., and Zaina, L. (2022). Towards understanding how software startups deal with ux from customer and user information. In International Conference on Software Business, pages 287–303. Springer.
Cretu, L.-G. (2012). Smart cities design using event-driven paradigm and semantic web. Informatica Economica, 16(4):57.
Deininger, M. E., von der Grün, M., Piepereit, R., Schneider, S., Santhanavanich, T., Coors, V., and Voß, U. (2020). A continuous, semi-automated workflow: from 3d city models with geometric optimization and cfd simulations to visualization of wind in an urban environment. ISPRS International Journal of Geo-Information, 9(11):657.
Díaz-Oreiro, I., López, G., Quesada, L., and Guerrero, L. A. (2021). Ux evaluation with standardized questionnaires in ubiquitous computing and ambient intelligence: a systematic literature review. Advances in Human-Computer Interaction, 2021:1–22.
Gubbi, J., Buyya, R., Marusic, S., and Palaniswami, M. (2013). Internet of things (iot): A vision, architectural elements, and future directions. Future generation computer systems, 29(7):1645–1660.
Hacid, H., Al-Shamaileh, O., and Maamar, Z. (2023). An exploratory study of rethinking user experience in the internet of things. Computing, 105(1):1–12.
Hart, S. G. and Staveland, L. E. (1988). Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Advances in psychology, volume 52, pages 139–183. Elsevier.
Hassenzahl, M., Burmester, M., and Koller, F. (2003). Attrakdiff: Ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualität. Mensch & Computer 2003: Interaktion in Bewegung, pages 187–196.
Hassenzahl, M. and Tractinsky, N. (2006). User experience-a research agenda. Behaviour & information technology, 25(2):91–97.
Hidalgo, A., Serif, T., Groenli, T.-M., and Ghinea, G. (2022). On user experience in the internet of things. In 2022 IEEE International Conference on Internet of Things and Intelligence Systems (IoTaIS), pages 31–37. IEEE.
Ismail, N. A., Majid, N., and Hassan, S. A. (2019). Iot-based smart solid waste management system a systematic literature review. International Journal of Innovative Technology and Exploring Engineering, 8(8):1456–1462.
ISO9241-210 (2019). Ergonomics of human-system interaction—part 210: Human-centred design for interactive systems. ISO: Geneva, Switzerland.
Laugwitz, B., Held, T., and Schrepp, M. (2008). Construction and evaluation of a user experience questionnaire. In HCI and Usability for Education and Work: 4th Symposium of the Workgroup Human-Computer Interaction and Usability Engineering of the Austrian Computer Society, USAB 2008, Graz, Austria, November 20-21, 2008. Proceedings 4, pages 63–76. Springer.
Lee, L. N. and Kim, M. J. (2020). A critical review of smart residential environments for older adults with a focus on pleasurable experience. Frontiers in psychology, 10:3080.
Melo, B., Andrade, R., and Darin, T. (2022). Longitudinal user experience studies in the iot domain: a brief panorama and challenges to overcome. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems, pages 1–13.
Mendoza, Y. L. M., Duarte, E. F., de Queiroz, M. J. N., and Baranauskas, M. C. C. (2023). Evaluation in scenarios of ubiquity of technology: a systematic literature review on interactive installations. Personal and Ubiquitous Computing, 27(2):343–361.
Minge, M., Thüring, M., Wagner, I., and Kuhr, C. V. (2017). The mecue questionnaire: a modular tool for measuring user experience. In Advances in Ergonomics Modeling, Usability & Special Populations: Proceedings of the AHFE 2016 International Conference on Ergonomics Modeling, Usability & Special Populations, July 27-31, 2016, Walt Disney World®, Florida, USA, pages 115–128. Springer.
Motta, R. C., de Oliveira, K. M., and Travassos, G. H. (2018). On challenges in engineering iot software systems. In Proceedings of the XXXII Brazilian symposium on software engineering, pages 42–51.
Motta, R. C., de Oliveira, K. M., and Travassos, G. H. (2023). An evidence-based roadmap for iot software systems engineering. Journal of Systems and Software, 201:111680.
Nielsen, J. (1994). 10 usability heuristics for user interface design. [link]. Online accessed 08 Dec 2023.
Norman, D. and Nielsen, J. (2020). The definition of user experience (ux). [link]. Online accessed 08 May 2023.
Ntoa, S., Margetis, G., Antona, M., and Stephanidis, C. (2021). User experience evaluation in intelligent environments: A comprehensive framework. Technologies, 9(2):41.
Reinfurt, L., Breitenbücher, U., Falkenthal, M., Leymann, F., and Riegg, A. (2017). Internet of things patterns for devices: Powering, operating, and sensing. International Journal on Advances in Internet Technology Volume 10, Number 3 & 4, 2017.
Shin, D.-H. (2017). Conceptualizing and measuring quality of experience of the internet of things: Exploring how quality is perceived by users. Information & Management, 54(8):998–1011.
Stephanidis, C., Salvendy, G., Antona, M., Chen, J. Y., Dong, J., Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P., et al. (2019). Seven hci grand challenges. International Journal of Human–Computer Interaction, 35(14):1229–1269.
Syed, A. S., Sierra-Sosa, D., Kumar, A., and Elmaghraby, A. (2021). Iot in smart cities: A survey of technologies, practices and challenges. Smart Cities, 4(2):429–475.
Tricco, A. C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J., Perrier, L., Hutton, B., Moher, D., and Straus, S. E. (2015). A scoping review of rapid review methods. BMC medicine, 13(1):1–15.
Venkatesh, V., Morris, M. G., Davis, G. B., and Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS quarterly, pages 425–478.
Whaiduzzaman, M., Barros, A., Chanda, M., Barman, S., Sultana, T., Rahman, M. S., Roy, S., and Fidge, C. (2022). A review of emerging technologies for iot-based smart cities. Sensors, 22(23):9271.
Aceves Gutierrez, L. C., Martin Gutierrez, J., and Del-Rio-Guerra, M. S. (2019). Having a smarter city through digital urban interfaces: an evaluation method. Applied Sciences, 9(17):3498.
Almeida, R. L., Darin, T. G., Andrade, R. M., and de Araújo, I. L. (2018). Towards developing a practical tool to assist ux evaluation in the iot scenario. In Anais Estendidos do XXIV Simpósio Brasileiro de Sistemas Multimídia e Web, pages 91–95. SBC.
Andrade, R., Carvalho, R. M., Araújo, I. L. d., Oliveira, K. M., and Maia, M. E. (2017). What changes from ubiquitous computing to internet of things in interaction evaluation? In Int. Conf. on Distributed, Ambient, and Pervasive Interactions, pages 3–21.
Angeloni, M. P. C., Oliveira, K. M. d., Grislin-Le Strugeon, E., Duque, R., and Tirnauca, C. (2023). A review on quality in use evaluation of smart environment applications: What’s next? In Companion Proceedings of the 2023 ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pages 9–15.
Brooke, J. (2013). Sus: a retrospective. j. usability stud. 8 (2), 29–40.
Cartaxo, B., Pinto, G., and Soares, S. (2018). The role of rapid reviews in supporting decision-making in software engineering practice. In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018, pages 24–34.
Choma, J., Machado, L., de Souza, C. R., Sharp, H., Barroca, L., and Zaina, L. (2022). Towards understanding how software startups deal with ux from customer and user information. In International Conference on Software Business, pages 287–303. Springer.
Cretu, L.-G. (2012). Smart cities design using event-driven paradigm and semantic web. Informatica Economica, 16(4):57.
Deininger, M. E., von der Grün, M., Piepereit, R., Schneider, S., Santhanavanich, T., Coors, V., and Voß, U. (2020). A continuous, semi-automated workflow: from 3d city models with geometric optimization and cfd simulations to visualization of wind in an urban environment. ISPRS International Journal of Geo-Information, 9(11):657.
Díaz-Oreiro, I., López, G., Quesada, L., and Guerrero, L. A. (2021). Ux evaluation with standardized questionnaires in ubiquitous computing and ambient intelligence: a systematic literature review. Advances in Human-Computer Interaction, 2021:1–22.
Gubbi, J., Buyya, R., Marusic, S., and Palaniswami, M. (2013). Internet of things (iot): A vision, architectural elements, and future directions. Future generation computer systems, 29(7):1645–1660.
Hacid, H., Al-Shamaileh, O., and Maamar, Z. (2023). An exploratory study of rethinking user experience in the internet of things. Computing, 105(1):1–12.
Hart, S. G. and Staveland, L. E. (1988). Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Advances in psychology, volume 52, pages 139–183. Elsevier.
Hassenzahl, M., Burmester, M., and Koller, F. (2003). Attrakdiff: Ein fragebogen zur messung wahrgenommener hedonischer und pragmatischer qualität. Mensch & Computer 2003: Interaktion in Bewegung, pages 187–196.
Hassenzahl, M. and Tractinsky, N. (2006). User experience-a research agenda. Behaviour & information technology, 25(2):91–97.
Hidalgo, A., Serif, T., Groenli, T.-M., and Ghinea, G. (2022). On user experience in the internet of things. In 2022 IEEE International Conference on Internet of Things and Intelligence Systems (IoTaIS), pages 31–37. IEEE.
Ismail, N. A., Majid, N., and Hassan, S. A. (2019). Iot-based smart solid waste management system a systematic literature review. International Journal of Innovative Technology and Exploring Engineering, 8(8):1456–1462.
ISO9241-210 (2019). Ergonomics of human-system interaction—part 210: Human-centred design for interactive systems. ISO: Geneva, Switzerland.
Laugwitz, B., Held, T., and Schrepp, M. (2008). Construction and evaluation of a user experience questionnaire. In HCI and Usability for Education and Work: 4th Symposium of the Workgroup Human-Computer Interaction and Usability Engineering of the Austrian Computer Society, USAB 2008, Graz, Austria, November 20-21, 2008. Proceedings 4, pages 63–76. Springer.
Lee, L. N. and Kim, M. J. (2020). A critical review of smart residential environments for older adults with a focus on pleasurable experience. Frontiers in psychology, 10:3080.
Melo, B., Andrade, R., and Darin, T. (2022). Longitudinal user experience studies in the iot domain: a brief panorama and challenges to overcome. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems, pages 1–13.
Mendoza, Y. L. M., Duarte, E. F., de Queiroz, M. J. N., and Baranauskas, M. C. C. (2023). Evaluation in scenarios of ubiquity of technology: a systematic literature review on interactive installations. Personal and Ubiquitous Computing, 27(2):343–361.
Minge, M., Thüring, M., Wagner, I., and Kuhr, C. V. (2017). The mecue questionnaire: a modular tool for measuring user experience. In Advances in Ergonomics Modeling, Usability & Special Populations: Proceedings of the AHFE 2016 International Conference on Ergonomics Modeling, Usability & Special Populations, July 27-31, 2016, Walt Disney World®, Florida, USA, pages 115–128. Springer.
Motta, R. C., de Oliveira, K. M., and Travassos, G. H. (2018). On challenges in engineering iot software systems. In Proceedings of the XXXII Brazilian symposium on software engineering, pages 42–51.
Motta, R. C., de Oliveira, K. M., and Travassos, G. H. (2023). An evidence-based roadmap for iot software systems engineering. Journal of Systems and Software, 201:111680.
Nielsen, J. (1994). 10 usability heuristics for user interface design. [link]. Online accessed 08 Dec 2023.
Norman, D. and Nielsen, J. (2020). The definition of user experience (ux). [link]. Online accessed 08 May 2023.
Ntoa, S., Margetis, G., Antona, M., and Stephanidis, C. (2021). User experience evaluation in intelligent environments: A comprehensive framework. Technologies, 9(2):41.
Reinfurt, L., Breitenbücher, U., Falkenthal, M., Leymann, F., and Riegg, A. (2017). Internet of things patterns for devices: Powering, operating, and sensing. International Journal on Advances in Internet Technology Volume 10, Number 3 & 4, 2017.
Shin, D.-H. (2017). Conceptualizing and measuring quality of experience of the internet of things: Exploring how quality is perceived by users. Information & Management, 54(8):998–1011.
Stephanidis, C., Salvendy, G., Antona, M., Chen, J. Y., Dong, J., Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P., et al. (2019). Seven hci grand challenges. International Journal of Human–Computer Interaction, 35(14):1229–1269.
Syed, A. S., Sierra-Sosa, D., Kumar, A., and Elmaghraby, A. (2021). Iot in smart cities: A survey of technologies, practices and challenges. Smart Cities, 4(2):429–475.
Tricco, A. C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J., Perrier, L., Hutton, B., Moher, D., and Straus, S. E. (2015). A scoping review of rapid review methods. BMC medicine, 13(1):1–15.
Venkatesh, V., Morris, M. G., Davis, G. B., and Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS quarterly, pages 425–478.
Whaiduzzaman, M., Barros, A., Chanda, M., Barman, S., Sultana, T., Rahman, M. S., Roy, S., and Fidge, C. (2022). A review of emerging technologies for iot-based smart cities. Sensors, 22(23):9271.
Publicado
06/05/2024
Como Citar
CHOMA, Joelma; ZAINA, Luciana.
UX evaluation of IoT-based applications for Smart Cities: a rapid systematic review. In: CONGRESSO IBERO-AMERICANO EM ENGENHARIA DE SOFTWARE (CIBSE), 27. , 2024, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 136-150.
DOI: https://doi.org/10.5753/cibse.2024.28444.
