Which kind of feedback helps novice programming students? An exploratory study with multiple feedback types in an introductory Python course
Abstract
Understanding error messages issued by compilers and interpreters is often one of the barriers faced by novice programming students. Generally, error messages are standardized and independent of their audience, making them unsuitable for less experienced programmers. Furthermore, students often have difficulty to recognize when the proposed solution is correct. This paper presents an exploratory case study accomplished during a Python programming course for novice students using the PEEF learning environment with the aim of analyzing and evaluating feedback use and effectiveness. In the course, students had access to different feedback types: pre-registered enhanced error messages, unit tests and chat. The results show that students assimilated the content, felt more confident and familiar with problem-solving, used unit tests and read error messages. The study also demonstrated a predilection among students for enhanced error messages and unit testing. Unit tests were used in 83.6% of projects. From these, 90.5% of projects were successful in testing. Regarding the use of pre-registered enhanced messages, the study found coverage of 81.1% of the errors issued. Enhanced messages were used to solve the activities, but 54% of them did not help to solve the problem. On the other hand, chat was less explored by students. Finally, we found out that the use of tests and enhanced messages contributes positively to learning. However, the method of pre-registering messages is not flexible to different variations of errors, leading to low assertiveness.
References
Luis Gustavo J. Araujo, Roberto A. Bittencourt, and Christina F. G. Chavez. 2021. Python Enhanced Error Feedback: Uma IDE Online de Apoio ao Processo de Ensino-Aprendizagem em Programação. In Anais do Simpósio Brasileiro de Educação em Computação. SBC, 326–333.
Brett A Becker. 2015. An exploration of the effects of enhanced compiler error messages for computer programming novices. Ph.D. Dissertation. Technological University Dublin.
Brett A. Becker. 2016. A New Metric to Quantify Repeated Compiler Errors for Novice Programmers. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (Arequipa, Peru) (ITiCSE ’16). Association for Computing Machinery, New York, NY, USA, 296–301.
Brett A Becker, Graham Glanville, Ricardo Iwashima, Claire McDonnell, Kyle Goslin, and Catherine Mooney. 2016. Effective compiler error message enhancement for novice programming students. Computer Science Education 26, 2-3, 148–175.
Tessa Charles and Carl Gwilliam. 2023. The Effect of Automated Error Message Feedback on Undergraduate Physics Students Learning Python: Reducing Anxiety and Building Confidence. Journal for STEM Education Research 6, 1, 1–32.
Eryck Pedro da Silva, Ricardo Edgard Caceffo, and Rodolfo Azevedo. 2023. Passar nos casos de teste é suficiente? Identificação e análise de problemas de compreensão em códigos corretos. In Anais do III Simpósio Brasileiro de Educação em Computação. SBC, SBC, Porto Alegre, RS, Brasil, 119–129.
Galileu Santos de Jesus, Jaine da Conceição Santos, Kleber Tarcísio Oliveira Santos, and Alberto Costa Neto. 2019. Análise dos erros mais comuns de aprendizes de programação que utilizam a linguagem Python. Anais do Computer on the Beach 10, 1, 406–415.
Paul Denny, Andrew Luxton-Reilly, and Dave Carpenter. 2014. Enhancing syntax error messages appears ineffectual. In Proceedings of the 2014 conference on Innovation & technology in computer science education. Association for Computing Machinery, Nova York, Nova York, Estados Unidos, 273–278.
Harry Barton Essel, Dimitrios Vlachopoulos, Akosua Tachie-Menson, Esi Eduafua Johnson, and Papa Kwame Baah. 2022. The impact of a virtual teaching assistant (chatbot) on students’ learning in Ghanaian higher education. International Journal of Educational Technology in Higher Education 19, 1, 1–19.
Tobias Kohn and Bill Manaris. 2020. Tell Me What’s Wrong: A Python IDE with Error Messages. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE ’20). Association for Computing Machinery, New York, NY, USA, 1054–1060.
Rodziah Latih, Marini Abu Bakar, Norleyza Jailani, Noorazean Mohd Ali, Syahanim Mohd Salleh, and Abdullah Mohd Zin. 2017. PC 2 to support instant feedback and good programming practice. In 2017 6th International Conference on Electrical Engineering and Informatics (ICEEI). IEEE, IEEE, Langkawi, Malasia, 1–5.
Hamza Manzoor, Amit Naik, Clifford A Shaffer, Chris North, and Stephen H Edwards. 2020. Auto-grading jupyter notebooks. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. Association for Computing Machinery, Nova York, Nova York, Estados Unidos, 1139–1144.
David Pritchard. 2015. Frequency distribution of error messages. In Proceedings of the 6th Workshop on Evaluation and Usability of Programming Languages and Tools. Association for Computing Machinery, Nova York, Nova York, Estados Unidos, 1–8.
V Javier Traver. 2010. On compiler error messages: what they say and what they mean. Advances in Human-Computer Interaction 2010, 1–26.
Matthew Verleger and James Pembridge. 2018. A pilot study integrating an AI-driven chatbot in an introductory programming course. In 2018 IEEE frontiers in education conference (FIE). IEEE, IEEE, San Jose, California, Estados Unidos, 1–4.
Nick Webb, David Benyon, Jay Bradley, Preben Hansen, and Oli Mival. 2010. Wizard of Oz Experiments for a Companion Dialogue System: Eliciting Companionable Conversation. In International Conference on Language Resources and Evaluation. LREC, Valletta, Malta, 6.
Robert K Yin. 2001. Estudo de Caso-: Planejamento e métodos. Bookman editora, Porto Alegre, RS, Brasil.
