Proposal and Preliminary Evaluation of a Learner Experience Evaluation Model in Information Systems
Resumo
Context: Learner eXperience (LX) is defined as the perceptions, responses and performance of learners through interaction with Information Systems (ISs). The LX evaluation of ISs can be performed using different approaches and considering different elements. Problem: However, from a Systematic Mapping Study (SMS), it was noticed that there is no standardization of LX evaluation approaches and that consider as many elements as possible. In this way, it is important to evaluate LX holistically, considering the diversity of learners and their learning preferences so that all experience aspects are considered. Solution: Therefore, the Learner Experience Evaluation Model (LEEM) was proposed. LEEM aims to evaluate and improve the learner's experience using technological resources considering different LX forms and elements. IS Theory: For the LEEM proposal, the Design Theory was followed, focusing on the design of IS artifacts. Method: The research methodology used was Design Science Research (DSR). In the DSR Knowledge Cycle, an SMS was carried out; in the Design Cycle, LEEM was proposed; and in the Relevance Cycle, a preliminary assessment of LEEM with 3 experts was conducted. Summary of Results: In SMS, 54 LX elements and 32 LX approaches were identified, which served as the basis for the LEEM construction. In the preliminary evaluation, experts suggest improvements, such as the replacement of the Likert scale with pairs of words in step 1 of the LEEM model. Contributions and Impact in the IS area: The LEEM model contributes to the IS area, as it allows considering non-technical aspects in the evaluation to promote new interactions and experiences by interested parts.
Referências
Isabel Cafezeiro, José Viterbo, Leonardo Cruz da Costa, Luciana Salgado, Marcelo da Costa Rocha, and Rodrigo Salvador Monteiro. 2017. Strengthening of the sociotechnical approach in information systems research. Sociedade Brasileira de Computação (2017), 133–147.
Vanessa Camilleri, Sara de Freitas, Matthew Montebello, and Paul McDonagh-Smith. 2013. A case study inside virtual worlds: use of analytics for immersive spaces. In Proceedings of the Third International Conference on Learning Analytics and Knowledge. 230–234.
Jared R Chapman, Eugene L Seeley, Norman S Wright, Lowell M Glenn, and Lynn L Adams. 2016. An empirical evaluation of a broad ranging e-text adoption with recommendations for improving deployment success for students.e-Journal of Business Education and Scholarship of Teaching 10, 2 (2016), 1–14.
Fábio Aparecido Candido Da Silva, Marluce Rodrigues Pereira, and André Pimenta Freire. 2022. Understanding Interaction and Organizational Issues in Radiology Information Systems: A Qualitative Study with Health Professionals. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3535511.3535557
Gabriela C dos Santos, Deivid Eive dos S Silva, and Natasha MC Valentim. 2022. Um Mapeamento Sistemático da Literatura sobre Iniciativas que avaliam a Experiência do Aprendiz. In Anais do XXXIII Simpósio Brasileiro de Informática na Educação. SBC, 621–633.
Márcio Rocha Ferreira and Tadeu Moreira de Classe. 2022. Design de Pré-Roteiros para Jogos Digitais Baseados em Processos de Negócios. iSys-Brazilian Journal of Information Systems 15, 1 (2022), 8–1.
Panagiotis Fotaris, Theodoros Mastoras, Richard Leinfellner, and Yasmine Rosunally. 2016. Climbing up the leaderboard: An empirical study of applying gamification techniques to a computer programming class.Electronic Journal of e-learning 14, 2 (2016), 94–110.
Ronghuai Huang, J Michael Spector, and Junfeng Yang. 2019. Educational Technology a Primer for the 21st Century. Springer. 91–105 pages. https://doi.org/10.1007/978-981-13-6643-7
ISO9241-210. 2019. Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems. [link] [Online; acessado em 23/02/2022].
Atsuko Kawano, Yuji Motoyama, and Mikio Aoyama. 2019. A LX (Learner EXperience)-Based Evaluation Method of the Education and Training Programs for Professional Software Engineers. In Proceedings of the 2019 7th International Conference on Information and Education Technology (Aizu-Wakamatsu, Japan) (ICIET 2019). Association for Computing Machinery, New York, NY, USA, 151–159. https://doi.org/10.1145/3323771.3323789
Sari Kujala and Kaisa Väänänen-Vainio-Mattila. 2009. Value of information systems and products: Understanding the users’ perspective and values. Journal of Information Technology Theory and Application (JITTA) 9, 4 (2009), 4.
Peter Lang. 1980. Behavioral treatment and bio-behavioral assessment: Computer applications. Technology in mental health care delivery systems (1980), 119–137.
Marianne Lykke, Mayela Coto, Christian Jantzen, Sonia Mora, and Niels Vandel. 2015. Motivating Students through Positive Learning Experiences: A Comparison of Three Learning Designs for Computer Programming Courses.Journal of Problem Based Learning in Higher Education 3, 2 (2015), 80–108.
Nathan Magyar and Stephanie R. Haley. 2020. Balancing Learner Experience and User Experience in a Peer Feedback Web Application for MOOCs. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3375232
Silvana M Melo. 2009. Inspeção de software. University of São Paulo: São Carlos, SP (2009).
Jakob Nielsen and Robert L. Mack. 1994. Usability Inspection Methods. John Wiley & Sons, Inc., USA.
Eeva Nygren, A Seugnet Blignaut, Verona Leendertz, and Erkki Sutinen. 2019. Quantitizing affective data as project evaluation on the use of a mathematics mobile game and intelligent tutoring system. Informatics in Education 18, 2 (2019), 375–402.
Priscilla BB Pagliusoa, Claudia de Andrade Tambasciaa, André Villas-Boasa, Miriam Ellen de Freitasa, and Maria Fernanda V Levantezia. 2003. GVR–Guia de Validação de Requisitos baseados nas técnicas PBR e ad-hoc resultante de um estudo de caso no CPqD1. In WORKSHOP EM ENGENHARIA DE REQUISITOS, Vol. 3. Piracicaba, SP, Brasil, 183–197.
Mariano Pimentel, Denise Filippo, and Flávia Maria Santoro. 2019. Design Science Research: fazendo pesquisas científicas rigorosas atreladas ao desenvolvimento de artefatos computacionais projetados para a educação. Metodologia de Pesquisa em Informática na Educação: Concepção da Pesquisa. Porto Alegre: SBC (2019), 5–29.
Jenny Ruiz and Monique Snoeck. 2018. Adapting Kirkpatrick's evaluation model to technology enhanced learning. In Proceedings of the 21st ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings. 135–142.
Lei Shi. 2014. Defining and Evaluating Learner Experience for Social Adaptive E-Learning. In 2014 Imperial College Computing Student Workshop(OpenAccess Series in Informatics (OASIcs), Vol. 43), Rumyana Neykova and Nicholas Ng (Eds.). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 74–82. https://doi.org/10.4230/OASIcs.ICCSW.2014.74
Forrest Shull, Manoel G Mendoncça, Victor Basili, Jeffrey Carver, José C Maldonado, Sandra Fabbri, Guilherme Horta Travassos, and Maria Cristina Ferreira. 2004. Knowledge-sharing issues in experimental software engineering. Empirical Software Engineering 9, 1 (2004), 111–137.
