Feasibility Study of a Model that evaluates the Learner Experience: A Quantitative and Qualitative Analysis
Resumo
Learner eXperience (LX) is a concept derived from User eXperience (UX) and it can be defined as the perceptions, answers, and performances of learners interacting with learning environments, educational products, and resources. Evaluating the LX to obtain experiences that support and facilitate learning and knowledge mastery is important. Thus, we developed the LEEM to assess and improve the learner’s experience using Digital Communication and Information Technologies during learning. The LEEM is a generic evaluation model and can be used for any level of education; it can be worked independently of the discipline and used with any educational technology. Therefore, this paper presents a feasibility study to evaluate the LEEM steps and sentences from the perspective of potential users. Nineteen teachers from different levels of education participated in this study. The study results were analyzed and generated in a new version of LEEM. The results showed positive points of LEEM, such as a practical, objective, easy-to-use, and useful model. In addition, opportunities for improving some items and sentence of LEEM was obtained. The teachers also suggested adding a description at the ends of the scales to facilitate the response to the items. This study contributes to creating a body of knowledge about LEEM, analyzing its use feasibility and evolution.
Palavras-chave:
Learner eXperience, evaluation, model, LX, qualitative analysis, quantitative analysis, student experience
Referências
Juliet Corbin 1990. Basics of qualitative research grounded theory procedures and techniques. [link]
Claiton Marques Correa, Gabriel Viegas Maciel de Freitas, André Luis dos Santos Eberhardt, and Milene Selbach Silveira. 2021. From Now on: Experiences from User-Based Research in Remote Settings. In Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems (Virtual Event, Brazil) (IHC ’21). Association for Computing Machinery, New York, NY, USA, Article 25, 7 pages. DOI: 10.1145/3472301.3484334
Elton José da Silva and Hugo Eduardo Ziviani. 2018. Desenho e Música no Ensino de IHC: relato de experiência de uma aula sobre conceitos básicos da Engenharia Semiótica. In Anais Estendidos do XVII Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (Belém). SBC, Porto Alegre, RS, Brasil. DOI: 10.5753/ihc.2018.4210
Gabriela Corbari dos Santos, Deivid Eive dos S. Silva, and Natasha Malveira C. Valentim. 2023. Proposal and Preliminary Evaluation of a Learner Experience Evaluation Model in Information Systems. In Proceedings of the XIX Brazilian Symposium on Information Systems (Maceió, Brazil) (SBSI ’23). Association for Computing Machinery, New York, NY, USA, 308–316. DOI: 10.1145/3592813.3592919
Panagiotis Fotaris, Theodoros Mastoras, Richard Leinfellner, and Yasmine Rosunally. 2016. Climbing up the leaderboard: An empirical study of applying gamification techniques to a computer programming class. Electronic Journal of e-learning 14, 2 (2016), 94–110. [link]
Ronghuai Huang, J Michael Spector, and Junfeng Yang. 2019. Educational Technology a Primer for the 21st Century. Springer, Singapore. DOI: 10.1007/978-981-13-6643-7
Rafael Eiki Matheus Imamura and Maria Cecília Calani Baranauskas. 2019. A Framework for Socio-Enactive Educational Systems: Linking Learning, Design, and Technology. In IHC ’19: XVIII Brazilian Symposium on Human Factors in Computing Systems (Vitória, Espírito Santo, Brazil) (IHC ’19). Association for Computing Machinery, New York, NY, USA, Article 1, 11 pages. DOI: 10.1145/3357155.3358443
ISO9241-210. 2019. Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems. [link] [Online; acessado em 23/02/2022].
Atsuko Kawano, Yuji Motoyama, and Mikio Aoyama. 2019. A LX (Learner EXperience)-Based Evaluation Method of the Education and Training Programs for Professional Software Engineers. In Proceedings of the 2019 7th International Conference on Information and Education Technology (Aizu-Wakamatsu, Japan) (ICIET 2019). Association for Computing Machinery, New York, NY, USA, 151–159. DOI: 10.1145/3323771.3323789
Peter Lang. 1980. Behavioral treatment and bio-behavioral assessment: Computer applications., 119–137 pages. [link]
Danilo Teixeira Lima, Rodrigo Oliveira Zacarias, Kennedy Edson Silva de Souza, Rodrigo Pereira dos Santos, and Marcos César da Rocha Seruffo. 2021. Analytical Model for Classifying Areas of Interest in Interactive Systems. In Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems (Virtual Event, Brazil) (IHC ’21). Association for Computing Machinery, New York, NY, USA, Article 4, 6 pages. DOI: 10.1145/3472301.3484357
Suéllen R. Martinelli and Luciana A. M. Zaina. 2021. Learning HCI from a Virtual Flipped Classroom: Improving the Students’ Experience in Times of COVID-19. In Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems (Virtual Event, Brazil) (IHC ’21). Association for Computing Machinery, New York, NY, USA, Article 34, 11 pages. DOI: 10.1145/3472301.3484326
Luã Marcelo Muriana and Maria Cecília Calani Baranauskas. 2021. Affecting User’s Self-Esteem: Analysis under the Self-Determination Theory Perspective and Design Recommendations. In Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems (Virtual Event, Brazil) (IHC ’21). Association for Computing Machinery, New York, NY, USA, Article 1, 12 pages. DOI: 10.1145/3472301.3484331
Jean Rosa, Beatriz do Rêgo, Filipe Garrido, Pedro Valente, Nuno Nunes, and Ecivaldo Matos. 2020. Interaction Design and Requirements Elicitation Integrated through SPIDe: a feasibility study. In Anais do XIX Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (Evento Online). SBC, Porto Alegre, RS, Brasil, 211–220. [link]
Jenny Ruiz and Monique Snoeck. 2018. Adapting Kirkpatrick’s Evaluation Model to Technology Enhanced Learning. In MODELS ’18: ACM/IEEE 21th International Conference on Model Driven Engineering Languages and Systems (Copenhagen, Denmark) (MODELS ’18). Association for Computing Machinery, New York, NY, USA, 135–142. DOI: 10.1145/3270112.3270114
Gabriela Santos, Deivid Silva, and Natasha Valentim. 2022. Um Mapeamento Sistemático da Literatura sobre Iniciativas que avaliam a Experiência do Aprendiz. In Anais do XXXIII Simpósio Brasileiro de Informática na Educação (Manaus). SBC, Porto Alegre, RS, Brasil, 621–633. DOI: 10.5753/sbie.2022.224673
Lei Shi. 2014. Defining and Evaluating Learner Experience for Social Adaptive E-Learning. In 2014 Imperial College Computing Student Workshop(OpenAccess Series in Informatics (OASIcs), Vol. 43), Rumyana Neykova and Nicholas Ng (Eds.). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 74–82. DOI: 10.4230/OASIcs.ICCSW.2014.74
Forrest Shull, Manoel G Mendoncça, Victor Basili, Jeffrey Carver, José C Maldonado, Sandra Fabbri, Guilherme Horta Travassos, and Maria Cristina Ferreira. 2004. Knowledge-sharing issues in experimental software engineering. Empirical Software Engineering 9, 1 (2004), 111–137. DOI: 10.1023/B:EMSE.0000013516.80487.33
Elliot Soloway, Mark Guzdial, and Kenneth E. Hay. 1994. Learner-Centered Design: The Challenge for HCI in the 21st Century. Interactions 1, 2 (apr 1994), 36–48. DOI: 10.1145/174809.174813
Marta S. Tabares, Paola Vallejo, Alex Montoya, Jose Sanchez, and Daniel Correa. 2021. SECA: A Feedback Rules Model in a Ubiquitous Microlearning Context. In DATA’21: International Conference on Data Science, E-learning and Information Systems 2021 (Ma’an, Jordan) (DATA’21). Association for Computing Machinery, New York, NY, USA, 136–142. DOI: 10.1145/3460620.3460745
Viswanath Venkatesh and Hillol Bala. 2008. Technology acceptance model 3 and a research agenda on interventions. Decision sciences 39, 2 (2008), 273–315. DOI: 10.1111/j.1540-5915.2008.00192.x
Benigna Maria de Freitas Villas Boas. 2006. Avaliação formativa e formaçãoo de professores: ainda um desafio. Linhas Críticas 12 (06 2006), 75 – 90. [link]
Claes Wholin, Per Runeson, Martin Host, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering: an introduction., 236 pages. DOI: 10.1007/978-3-642-29044-2
Claiton Marques Correa, Gabriel Viegas Maciel de Freitas, André Luis dos Santos Eberhardt, and Milene Selbach Silveira. 2021. From Now on: Experiences from User-Based Research in Remote Settings. In Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems (Virtual Event, Brazil) (IHC ’21). Association for Computing Machinery, New York, NY, USA, Article 25, 7 pages. DOI: 10.1145/3472301.3484334
Elton José da Silva and Hugo Eduardo Ziviani. 2018. Desenho e Música no Ensino de IHC: relato de experiência de uma aula sobre conceitos básicos da Engenharia Semiótica. In Anais Estendidos do XVII Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (Belém). SBC, Porto Alegre, RS, Brasil. DOI: 10.5753/ihc.2018.4210
Gabriela Corbari dos Santos, Deivid Eive dos S. Silva, and Natasha Malveira C. Valentim. 2023. Proposal and Preliminary Evaluation of a Learner Experience Evaluation Model in Information Systems. In Proceedings of the XIX Brazilian Symposium on Information Systems (Maceió, Brazil) (SBSI ’23). Association for Computing Machinery, New York, NY, USA, 308–316. DOI: 10.1145/3592813.3592919
Panagiotis Fotaris, Theodoros Mastoras, Richard Leinfellner, and Yasmine Rosunally. 2016. Climbing up the leaderboard: An empirical study of applying gamification techniques to a computer programming class. Electronic Journal of e-learning 14, 2 (2016), 94–110. [link]
Ronghuai Huang, J Michael Spector, and Junfeng Yang. 2019. Educational Technology a Primer for the 21st Century. Springer, Singapore. DOI: 10.1007/978-981-13-6643-7
Rafael Eiki Matheus Imamura and Maria Cecília Calani Baranauskas. 2019. A Framework for Socio-Enactive Educational Systems: Linking Learning, Design, and Technology. In IHC ’19: XVIII Brazilian Symposium on Human Factors in Computing Systems (Vitória, Espírito Santo, Brazil) (IHC ’19). Association for Computing Machinery, New York, NY, USA, Article 1, 11 pages. DOI: 10.1145/3357155.3358443
ISO9241-210. 2019. Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems. [link] [Online; acessado em 23/02/2022].
Atsuko Kawano, Yuji Motoyama, and Mikio Aoyama. 2019. A LX (Learner EXperience)-Based Evaluation Method of the Education and Training Programs for Professional Software Engineers. In Proceedings of the 2019 7th International Conference on Information and Education Technology (Aizu-Wakamatsu, Japan) (ICIET 2019). Association for Computing Machinery, New York, NY, USA, 151–159. DOI: 10.1145/3323771.3323789
Peter Lang. 1980. Behavioral treatment and bio-behavioral assessment: Computer applications., 119–137 pages. [link]
Danilo Teixeira Lima, Rodrigo Oliveira Zacarias, Kennedy Edson Silva de Souza, Rodrigo Pereira dos Santos, and Marcos César da Rocha Seruffo. 2021. Analytical Model for Classifying Areas of Interest in Interactive Systems. In Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems (Virtual Event, Brazil) (IHC ’21). Association for Computing Machinery, New York, NY, USA, Article 4, 6 pages. DOI: 10.1145/3472301.3484357
Suéllen R. Martinelli and Luciana A. M. Zaina. 2021. Learning HCI from a Virtual Flipped Classroom: Improving the Students’ Experience in Times of COVID-19. In Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems (Virtual Event, Brazil) (IHC ’21). Association for Computing Machinery, New York, NY, USA, Article 34, 11 pages. DOI: 10.1145/3472301.3484326
Luã Marcelo Muriana and Maria Cecília Calani Baranauskas. 2021. Affecting User’s Self-Esteem: Analysis under the Self-Determination Theory Perspective and Design Recommendations. In Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems (Virtual Event, Brazil) (IHC ’21). Association for Computing Machinery, New York, NY, USA, Article 1, 12 pages. DOI: 10.1145/3472301.3484331
Jean Rosa, Beatriz do Rêgo, Filipe Garrido, Pedro Valente, Nuno Nunes, and Ecivaldo Matos. 2020. Interaction Design and Requirements Elicitation Integrated through SPIDe: a feasibility study. In Anais do XIX Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (Evento Online). SBC, Porto Alegre, RS, Brasil, 211–220. [link]
Jenny Ruiz and Monique Snoeck. 2018. Adapting Kirkpatrick’s Evaluation Model to Technology Enhanced Learning. In MODELS ’18: ACM/IEEE 21th International Conference on Model Driven Engineering Languages and Systems (Copenhagen, Denmark) (MODELS ’18). Association for Computing Machinery, New York, NY, USA, 135–142. DOI: 10.1145/3270112.3270114
Gabriela Santos, Deivid Silva, and Natasha Valentim. 2022. Um Mapeamento Sistemático da Literatura sobre Iniciativas que avaliam a Experiência do Aprendiz. In Anais do XXXIII Simpósio Brasileiro de Informática na Educação (Manaus). SBC, Porto Alegre, RS, Brasil, 621–633. DOI: 10.5753/sbie.2022.224673
Lei Shi. 2014. Defining and Evaluating Learner Experience for Social Adaptive E-Learning. In 2014 Imperial College Computing Student Workshop(OpenAccess Series in Informatics (OASIcs), Vol. 43), Rumyana Neykova and Nicholas Ng (Eds.). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 74–82. DOI: 10.4230/OASIcs.ICCSW.2014.74
Forrest Shull, Manoel G Mendoncça, Victor Basili, Jeffrey Carver, José C Maldonado, Sandra Fabbri, Guilherme Horta Travassos, and Maria Cristina Ferreira. 2004. Knowledge-sharing issues in experimental software engineering. Empirical Software Engineering 9, 1 (2004), 111–137. DOI: 10.1023/B:EMSE.0000013516.80487.33
Elliot Soloway, Mark Guzdial, and Kenneth E. Hay. 1994. Learner-Centered Design: The Challenge for HCI in the 21st Century. Interactions 1, 2 (apr 1994), 36–48. DOI: 10.1145/174809.174813
Marta S. Tabares, Paola Vallejo, Alex Montoya, Jose Sanchez, and Daniel Correa. 2021. SECA: A Feedback Rules Model in a Ubiquitous Microlearning Context. In DATA’21: International Conference on Data Science, E-learning and Information Systems 2021 (Ma’an, Jordan) (DATA’21). Association for Computing Machinery, New York, NY, USA, 136–142. DOI: 10.1145/3460620.3460745
Viswanath Venkatesh and Hillol Bala. 2008. Technology acceptance model 3 and a research agenda on interventions. Decision sciences 39, 2 (2008), 273–315. DOI: 10.1111/j.1540-5915.2008.00192.x
Benigna Maria de Freitas Villas Boas. 2006. Avaliação formativa e formaçãoo de professores: ainda um desafio. Linhas Críticas 12 (06 2006), 75 – 90. [link]
Claes Wholin, Per Runeson, Martin Host, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering: an introduction., 236 pages. DOI: 10.1007/978-3-642-29044-2
Publicado
16/10/2023
Como Citar
CORBARI DOS SANTOS, Gabriela; DOS S. SILVA, Deivid Eive; M. C. VALENTIM, Natasha.
Feasibility Study of a Model that evaluates the Learner Experience: A Quantitative and Qualitative Analysis. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 22. , 2023, Maceió/AL.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
