Applying Design Science Research to develop a Technique to Evaluate the Usability and User eXperience of Learning Management Systems
Resumo
As Internet-based technology becomes more accessible, learning institutions have been adopting platforms such as Learning Management Systems (LMSs) to support and also complement face to face learning. Their widespread adoption raises the importance to evaluate them regarding quality in use attributes such as usability and User eXperience (UX). This paper presents the application of Design Science Research (DSR) to develop a technique and a tool to evaluate the usability and UX of LMSs called TUXEL. By applying DSR, we ensure a high level of rigor in the development process, combining theory and practice. The empirical studies indicated that TUXEL is adequate, in addition to providing better results compared to other techniques.
Palavras-chave:
usability, user experience, LMS, Design Science Research, usability evaluation
Referências
Gordillo, A., Barra, E., Aguirre, S., and Quemada, J. (2014). The usefulness of usability and user experience evaluation methods on an e-Learning platform development from a developer's perspective: A case study. In Frontiers in Education Conference (FIE), IEEE, p. 1-8.
Harrati, N., Bouchrika, I., Tari, A. and Ladjailia, A. (2016). Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. In Computers in Human Behavior, 61, p. 463-471.
Hevner, A. and Chatterjee, S. (2010). Design Research in Information Systems: Theory and Practice. In Integrated Series in Information Systems, v. 22, Springer.
Hovde, M. R. (2015). Effective user experience in online technical communication courses: employing multiple methods within organizational contexts to assess usability. In Proceedings of the 33rd Annual International Conference on the Design of Communication, ACM, p. 30.
ISO 9241-11 (2017). Ergonomics of human-system interaction - Part 11: Usability: Definitions and concepts.
ISO 9241-210 (2010). Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems.
Lanzilotti, R., Ardito, C., Costabile, M. F. and De Angeli, A. (2006). eLSE methodology: a systematic approach to the e-learning systems evaluation. In Journal of Educational Technology & Society, 9(4).
Laugwitz, B., Held, T. and Schrepp, M. (2008). Construction and evaluation of a user experience questionnaire. In Symposium of the Austrian HCI and Usability Engineering Group, Springer Berlin Heidelberg, p. 63-76.
Mtebe, J. S. and Kissaka, M. M. (2015). Heuristics for evaluating usability of Learning Management Systems in Africa. In IST-Africa Conference, 2015, IEEE, p. 1-13.
Nokelainen, P. (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students. In Journal of Educational Technology & Society, 9(2).
Oztekin, A., Kong, Z. J. and Uysal, O. (2010). UseLearn: A novel checklist and usability evaluation method for eLearning systems by criticality metric analysis. In International Journal of Industrial Ergonomics, 40(4), p. 455-469.
Nakamura, W. T. (2018). TUXEL: A Technique for User eXperience Evaluation in e-Learning. Master's thesis. Federal University of Amazonas, Manaus.
Nakamura, W. T., de Oliveira, E. H. T., and Conte, T. (2017a). Usability and User Experience evaluation of Learning Management Systems: A systematic mapping study. In Proceedings of the 19th International Conference on Enterprise Information Systems (ICEIS), 10(1), 97-108.
Nakamura, W., Marques, L., Rivero, L., Oliveira, E., and Conte, T. (2017b). Are Generic UX Evaluation Techniques Enough? A study on the UX Evaluation of the Edmodo Learning Management System. In Brazilian Symposium on Computers in Education (Vol. 28, No. 1, p. 1007).
Storey, M. A., Engström, E., Höst, M., Runeson, P. and Bjarnason, E. (2017). Using a Visual Abstract as a Lens for Communicating and Promoting Design Science Research in Software Engineering. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, p. 181-186.
Theng, Y. L. and Sin, J. (2012). Evaluating usability and efficaciousness of an e-learning system: A quantitative, model-driven approach. In Advanced Learning Technologies (ICALT), 2012 IEEE 12th International Conference on, IEEE, p. 303-307.
Van Schaik, P. and Ling, J. (2010). An integrated model of interaction experience for information retrieval in a Web-based encyclopaedia. In Interacting with Computers, 23(1), 18-32.
Wieringa, R. (2014). Design Science Methodology for Information Systems and Software Engineering. Springer-Verlag Berlin Heidelberg.
Zaharias, P. and Pappas, C. (2016). Quality Management of Learning Management Systems: A User Experience Perspective. In Current Issues in Emerging eLearning, 3(1), 5.
Harrati, N., Bouchrika, I., Tari, A. and Ladjailia, A. (2016). Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. In Computers in Human Behavior, 61, p. 463-471.
Hevner, A. and Chatterjee, S. (2010). Design Research in Information Systems: Theory and Practice. In Integrated Series in Information Systems, v. 22, Springer.
Hovde, M. R. (2015). Effective user experience in online technical communication courses: employing multiple methods within organizational contexts to assess usability. In Proceedings of the 33rd Annual International Conference on the Design of Communication, ACM, p. 30.
ISO 9241-11 (2017). Ergonomics of human-system interaction - Part 11: Usability: Definitions and concepts.
ISO 9241-210 (2010). Ergonomics of human-system interaction - Part 210: Human-centred design for interactive systems.
Lanzilotti, R., Ardito, C., Costabile, M. F. and De Angeli, A. (2006). eLSE methodology: a systematic approach to the e-learning systems evaluation. In Journal of Educational Technology & Society, 9(4).
Laugwitz, B., Held, T. and Schrepp, M. (2008). Construction and evaluation of a user experience questionnaire. In Symposium of the Austrian HCI and Usability Engineering Group, Springer Berlin Heidelberg, p. 63-76.
Mtebe, J. S. and Kissaka, M. M. (2015). Heuristics for evaluating usability of Learning Management Systems in Africa. In IST-Africa Conference, 2015, IEEE, p. 1-13.
Nokelainen, P. (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students. In Journal of Educational Technology & Society, 9(2).
Oztekin, A., Kong, Z. J. and Uysal, O. (2010). UseLearn: A novel checklist and usability evaluation method for eLearning systems by criticality metric analysis. In International Journal of Industrial Ergonomics, 40(4), p. 455-469.
Nakamura, W. T. (2018). TUXEL: A Technique for User eXperience Evaluation in e-Learning. Master's thesis. Federal University of Amazonas, Manaus.
Nakamura, W. T., de Oliveira, E. H. T., and Conte, T. (2017a). Usability and User Experience evaluation of Learning Management Systems: A systematic mapping study. In Proceedings of the 19th International Conference on Enterprise Information Systems (ICEIS), 10(1), 97-108.
Nakamura, W., Marques, L., Rivero, L., Oliveira, E., and Conte, T. (2017b). Are Generic UX Evaluation Techniques Enough? A study on the UX Evaluation of the Edmodo Learning Management System. In Brazilian Symposium on Computers in Education (Vol. 28, No. 1, p. 1007).
Storey, M. A., Engström, E., Höst, M., Runeson, P. and Bjarnason, E. (2017). Using a Visual Abstract as a Lens for Communicating and Promoting Design Science Research in Software Engineering. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, p. 181-186.
Theng, Y. L. and Sin, J. (2012). Evaluating usability and efficaciousness of an e-learning system: A quantitative, model-driven approach. In Advanced Learning Technologies (ICALT), 2012 IEEE 12th International Conference on, IEEE, p. 303-307.
Van Schaik, P. and Ling, J. (2010). An integrated model of interaction experience for information retrieval in a Web-based encyclopaedia. In Interacting with Computers, 23(1), 18-32.
Wieringa, R. (2014). Design Science Methodology for Information Systems and Software Engineering. Springer-Verlag Berlin Heidelberg.
Zaharias, P. and Pappas, C. (2016). Quality Management of Learning Management Systems: A User Experience Perspective. In Current Issues in Emerging eLearning, 3(1), 5.
Publicado
29/10/2018
Como Citar
NAKAMURA, Walter T.; OLIVEIRA, Elaine H. T. de; CONTE, Tayana.
Applying Design Science Research to develop a Technique to Evaluate the Usability and User eXperience of Learning Management Systems. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 29. , 2018, Fortaleza/CE.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2018
.
p. 953-962.
DOI: https://doi.org/10.5753/cbie.sbie.2018.953.
