Evolution of a Game Evaluation Model for Computer Science Education
Abstract
Educational games are assumed to be an efficient instructional strategy for computing education. However, it is essential to systematically evaluate such games in order to obtain sound evidence on their quality. A prominent model for the evaluation of educational games is MEEGA (Model for the Evaluation of Educational GAmes). However, analyzing the initial version we have identified limitations in terms of its validity and reliability. Therefore, the objective of this paper is to present an evolution of the model, MEEGA+. The MEEGA+ model has been systematically developed by decomposing evaluation goals into measures and defining measurement instruments to evaluate the perceived quality of educational games in terms of player experience and perceived learning from the viewpoint of the student and instructor. The model, thus, assists game creators, instructors and/or researchers to evaluate educational games and, hence, contribute to their improvement and adoption in practice.
References
ACM/IEEE-CS. (2013). Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science. Disponível em: [link] Acesso: 17 outubro 2016.
Anderson, L. W., Krathwohl, D. R., e Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives. Longman.
Basili, V. R., Caldiera, G., e Rombach, H. D. (1994). Goal, Question Metric Paradigm. In J. J.
Marciniak, Encyclopedia of Software Engineering. New York, USA.
Battistella, P. e Wangenheim, C. G. (2016). Games for Teaching Computing in Higher Education – A Systematic Review. In IEEE Technology and Engineering Education Journal, 9(1), 8-30.
Brooke, J. (1996). SUS-A quick and dirty usability scale. In Usability evaluation in industry, 189(194), 4-7.
Calderón A. e Ruiz M. (2015). A systematic literature review on serious games evaluation: An application to software project management. In Computers & Education, 87, 396-422.
Caulfield, C., Xia, J., Veal, D., e Maj, S. P. (2011). A systematic survey of games used for software engineering education. In Modern Applied Science, 5(6), 28-43.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, p. 319-340.
Dempsey, J., Rasmussen, K., e Lucassen, B. (1996). The instructional gaming literature: Implications and 99 sources. Technical Report 96-1. College of Education, University of South Alabama, USA.
DeVellis, R. F. (2016). Scale development: theory and applications. 4. ed. SAGE Publications.
Fu, F., Su, R., e Yu, S. (2009). EGameFlow: A scale to measure learners' enjoyment of elearning games. In Computers & Education, 52(1), 101-112.
Gámez, E. H. (2009). On the Core Elements of the Experience of Playing Video Games (Dissertation). UCL Interaction Centre, Department of Computer Science, London, UK.
Gresse von Wangenheim, C. e Shull, F. (2009). To Game or Not to Game? In IEEE Software, 26(2), 92-94.
Gresse von Wangenheim, C., Savi, R., e Borgatto, A. F. (2013). SCRUMIA - An educational game for teaching SCRUM in computing courses. In Journal of Systems and Software, 86(10), 2675-2687.
Instituto Brasileiro de Geografia e Estatística (IBGE). (2012). Censo Demográfico 2010: Características gerais da população, religião e pessoas com deficiência.
Keller, J. M. (1987). Development and Use of the ARCS Model of motivational Design. In Journal of Instructional Development, 10(3), 2-10.
Kirkpatrick, D. L. e Kirkpatrick, J. D. (2006). Evaluating training programs: the four levels, 3. ed. Berrett-Koehler Publishers, USA.
Kitchenham, B. (2010). Systematic literature reviews in software engineering – A tertiary study. Information and Software Technology, v. 52, n. 1, p. 792-805.
Krathwohl, D. R.; Bloom, B. S.; e Masia, B. B. (1973). Taxonomy of Educational Objectives, the Classification of Educational Goals. Handbook II: Affective Domain. Nova York, EUA, 1973.
O’Brien, H. L. e Toms, E. G. (2010). The Development and Evaluation of a Survey to Measure User Engagement. In Journal of the American Society for Information Science and Technology, 61(1), 50–69.
Petri, G. e Gresse von Wangenheim, C. (2016). How to Evaluate Educational Games: a Systematic Literature Review. In Journal of Universal Computer Science, 22(7), 992-1021.
Petri, G. e Gresse von Wangenheim, C. (2017). How games for computing education are evaluated: a systematic literature review. In Computers & Education, 107, 68-90.
Petri, G.; Gresse von Wangenheim, C., e Borgatto, A. F. (2016). MEEGA+: An Evolution of a Model of Educational Games. Relatório Técnico INCoD/GQS.03.2016.E. INCoD/INE/UFSC, Florianópolis, Brazil.
Petri, G.; Gresse von Wangenheim, C., e Borgatto, A. F. (2017). A large-scale evaluation of a model for the evaluation of games for teaching software engineering. Proc. of the 39th Int. Conf. on Software Engineering. Buenos Aires, Argentina.
Poels, K., Kort, Y. D., e Ijsselsteijn, W. (2007). It is always a lot of fun!: exploring dimensions of digital game experience using focus group methodology. Proc. of Conf. on Future Play, Toronto, Canada.
Savi, R., Gresse von Wangenheim, C., e Borgatto, A. F. (2011). A Model for the Evaluation of Educational Games for Teaching Software Engineering. Proc. of the 25th Simpósio Brasileiro de Engenharia de Software, São Paulo, Brazil.
Sindre, G. e Moody, D. (2003). Evaluating the Effectiveness of Learning Interventions: an Information Systems Case Study. Proc. of the 11th European Conf. on Information Systems, Naples, Italy.
Souza, M. e França, C. (2016). O que Explica o Sucesso de Jogos no Ensino de Engenharia de Software? Uma Teoria de Motivação. 24º Workshop sobre Educação em Computação, 2255-2264. Porto Alegre/RS, Brasil.
Sweetser, P. e Wyeth, P. (2005). GameFlow: a model for evaluating player enjoyment in games. In Computers in Entertainment, 3(3), 1-24.
Takatalo, J., Häkkinen, J., Kaistinen, J., e Nyman, G. (2010). Presence, Involvement, and Flow in Digital Games. In: Bernhaupt, R. (Ed.). Evaluating User Experience in Games: Concepts and Methods, Springer.
Tullis, T. e Albert, W. (2008). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan Kaufmann.
Wiebe, E. N., Lamb, A., Hardy, M., e Sharek, D. (2014). Measuring engagement in video gamebased environments: Investigation of the User Engagement Scale. In Computers in Human Behavior, 32, 123–132.
Yin, R.K. (2009). Case study research: design and methods, 4th ed. Sage Publications, USA.
Anderson, L. W., Krathwohl, D. R., e Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives. Longman.
Basili, V. R., Caldiera, G., e Rombach, H. D. (1994). Goal, Question Metric Paradigm. In J. J.
Marciniak, Encyclopedia of Software Engineering. New York, USA.
Battistella, P. e Wangenheim, C. G. (2016). Games for Teaching Computing in Higher Education – A Systematic Review. In IEEE Technology and Engineering Education Journal, 9(1), 8-30.
Brooke, J. (1996). SUS-A quick and dirty usability scale. In Usability evaluation in industry, 189(194), 4-7.
Calderón A. e Ruiz M. (2015). A systematic literature review on serious games evaluation: An application to software project management. In Computers & Education, 87, 396-422.
Caulfield, C., Xia, J., Veal, D., e Maj, S. P. (2011). A systematic survey of games used for software engineering education. In Modern Applied Science, 5(6), 28-43.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, p. 319-340.
Dempsey, J., Rasmussen, K., e Lucassen, B. (1996). The instructional gaming literature: Implications and 99 sources. Technical Report 96-1. College of Education, University of South Alabama, USA.
DeVellis, R. F. (2016). Scale development: theory and applications. 4. ed. SAGE Publications.
Fu, F., Su, R., e Yu, S. (2009). EGameFlow: A scale to measure learners' enjoyment of elearning games. In Computers & Education, 52(1), 101-112.
Gámez, E. H. (2009). On the Core Elements of the Experience of Playing Video Games (Dissertation). UCL Interaction Centre, Department of Computer Science, London, UK.
Gresse von Wangenheim, C. e Shull, F. (2009). To Game or Not to Game? In IEEE Software, 26(2), 92-94.
Gresse von Wangenheim, C., Savi, R., e Borgatto, A. F. (2013). SCRUMIA - An educational game for teaching SCRUM in computing courses. In Journal of Systems and Software, 86(10), 2675-2687.
Instituto Brasileiro de Geografia e Estatística (IBGE). (2012). Censo Demográfico 2010: Características gerais da população, religião e pessoas com deficiência.
Keller, J. M. (1987). Development and Use of the ARCS Model of motivational Design. In Journal of Instructional Development, 10(3), 2-10.
Kirkpatrick, D. L. e Kirkpatrick, J. D. (2006). Evaluating training programs: the four levels, 3. ed. Berrett-Koehler Publishers, USA.
Kitchenham, B. (2010). Systematic literature reviews in software engineering – A tertiary study. Information and Software Technology, v. 52, n. 1, p. 792-805.
Krathwohl, D. R.; Bloom, B. S.; e Masia, B. B. (1973). Taxonomy of Educational Objectives, the Classification of Educational Goals. Handbook II: Affective Domain. Nova York, EUA, 1973.
O’Brien, H. L. e Toms, E. G. (2010). The Development and Evaluation of a Survey to Measure User Engagement. In Journal of the American Society for Information Science and Technology, 61(1), 50–69.
Petri, G. e Gresse von Wangenheim, C. (2016). How to Evaluate Educational Games: a Systematic Literature Review. In Journal of Universal Computer Science, 22(7), 992-1021.
Petri, G. e Gresse von Wangenheim, C. (2017). How games for computing education are evaluated: a systematic literature review. In Computers & Education, 107, 68-90.
Petri, G.; Gresse von Wangenheim, C., e Borgatto, A. F. (2016). MEEGA+: An Evolution of a Model of Educational Games. Relatório Técnico INCoD/GQS.03.2016.E. INCoD/INE/UFSC, Florianópolis, Brazil.
Petri, G.; Gresse von Wangenheim, C., e Borgatto, A. F. (2017). A large-scale evaluation of a model for the evaluation of games for teaching software engineering. Proc. of the 39th Int. Conf. on Software Engineering. Buenos Aires, Argentina.
Poels, K., Kort, Y. D., e Ijsselsteijn, W. (2007). It is always a lot of fun!: exploring dimensions of digital game experience using focus group methodology. Proc. of Conf. on Future Play, Toronto, Canada.
Savi, R., Gresse von Wangenheim, C., e Borgatto, A. F. (2011). A Model for the Evaluation of Educational Games for Teaching Software Engineering. Proc. of the 25th Simpósio Brasileiro de Engenharia de Software, São Paulo, Brazil.
Sindre, G. e Moody, D. (2003). Evaluating the Effectiveness of Learning Interventions: an Information Systems Case Study. Proc. of the 11th European Conf. on Information Systems, Naples, Italy.
Souza, M. e França, C. (2016). O que Explica o Sucesso de Jogos no Ensino de Engenharia de Software? Uma Teoria de Motivação. 24º Workshop sobre Educação em Computação, 2255-2264. Porto Alegre/RS, Brasil.
Sweetser, P. e Wyeth, P. (2005). GameFlow: a model for evaluating player enjoyment in games. In Computers in Entertainment, 3(3), 1-24.
Takatalo, J., Häkkinen, J., Kaistinen, J., e Nyman, G. (2010). Presence, Involvement, and Flow in Digital Games. In: Bernhaupt, R. (Ed.). Evaluating User Experience in Games: Concepts and Methods, Springer.
Tullis, T. e Albert, W. (2008). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan Kaufmann.
Wiebe, E. N., Lamb, A., Hardy, M., e Sharek, D. (2014). Measuring engagement in video gamebased environments: Investigation of the User Engagement Scale. In Computers in Human Behavior, 32, 123–132.
Yin, R.K. (2009). Case study research: design and methods, 4th ed. Sage Publications, USA.
Published
2017-07-02
How to Cite
PETRI, Giani; VON WANGENHEIM, Christiane Gresse; BORGATTO, Adriano Ferreti.
Evolution of a Game Evaluation Model for Computer Science Education. In: WORKSHOP ON COMPUTING EDUCATION (WEI), 25. , 2017, São Paulo.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2017
.
p. 2327-2336.
ISSN 2595-6175.
DOI: https://doi.org/10.5753/wei.2017.3549.
