Is passing the test cases enough? Identification and analysis of misconceptions in correct code

• Eryck Pedro da Silva State University of Campinas http://orcid.org/0000-0001-5141-6936
• Ricardo Edgard Caceffo State University of Campinas https://orcid.org/0000-0001-9152-0658
• Rodolfo Azevedo State University of Campinas https://orcid.org/0000-0002-8803-0401

DOI: https://doi.org/10.5753/educomp.2023.228346

Abstract

The usage of automatic grading systems (autograders) supports introductory programming courses (CS1) teaching. However, the focus on correctness can obfuscate the verification of other issues present in code. In this work, it was investigated if code, deemed correct by an autograder, presented behaviors that could indicate issues in the understanding of concepts taught in CS1. These behaviors were denominated Misconceptions in Correct Code (MC³). By analyzing 2441 code, a list with 45 MC³ was developed and then assessed by CS1 instructors to identify which MC³ most need to be corrected in class and how this correction can be elaborated. In total, 15 MC³ were considered as the most severe and the instructors suggested automatic detection of MC³ and the use of Active Learning techniques in class. The obtained results can support the development of artifacts for interventions regarding MC³ in CS1.

Keywords: introductory programming, misconceptions, automatic grading, CS1

References

Vicki L Almstrum, Peter B Henderson, Valerie Harvey, Cinda Heeren, William Marion, Charles Riedesel, Leen-Kiat Soh, and Allison Elliott Tew. 2006. Concept inventories in computer science for the topic discrete mathematics. In Working group reports on ITiCSE on Innovation and technology in computer science education. 132–145.

Ada Araujo, Daniel Filho, Elaine Oliveira, Leandro Carvalho, Filipe Pereira, and David Oliveira. 2021. Mapeamento e análise empírica de misconceptions comuns em avaliações de introdução à programação. In Anais do Simpósio Brasileiro de Educação em Computação (On-line). SBC, Porto Alegre, RS, Brasil, 123–131.

Luis Gustavo Jesus Araujo, Roberto Almeida Bittencourt, and Christina von Flach Garcia Chavez. 2021. Python Enhanced Error Feedback: Uma IDE Online de Apoio ao Processo de Ensino-Aprendizagem em Programação. In Anais do Simpósio Brasileiro de Educação em Computação. SBC, 326–333.

Elisa Baniassad, Lucas Zamprogno, Braxton Hall, and Reid Holmes. 2021. STOP THE (AUTOGRADER) INSANITY: Regression Penalties to Deter Autograder Overreliance. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 1062–1068.

Brett A. Becker and Thomas Fitzpatrick. 2019. What do cs1 syllabi reveal about our expectations of introductory programming students?. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 1011–1017.

Brett A. Becker, Kyle Goslin, and Graham Glanville. 2018. The effects of enhanced compiler error messages on a syntax error debugging test. SIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on Computer Science Education 2018-Janua, 640–645.

C.C. Bonwell and J.A. Eison. 1991. Active Learning: Creating Excitement in the Classroom. Wiley. https://books.google.com.br/books?id=AW7uAAAAMAAJ

Ricardo Caceffo, Pablo Frank-Bolton, Renan Souza, and Rodolfo Azevedo. 2019. Identifying and validating java misconceptions toward a cs1 concept inventory. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education. 23–29.

Ricardo Caceffo, Guilherme Gama, and Rodolfo Azevedo. 2018. Exploring Active Learning Approaches to Computer Science Classes. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (Baltimore, Maryland, USA) (SIGCSE ’18). Association for Computing Machinery, New York, NY, USA, 922–927.

Ricardo Caceffo, SteveWolfman, Kellogg S. Booth, and Rodolfo Azevedo. 2016. Developing a Computer Science Concept Inventory for Introductory Programming. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education (Memphis, Tennessee, USA) (SIGCSE ’16). Association for Computing Machinery, New York, NY, USA, 364–369.

Andrew Cain and Muhammad Ali Babar. 2016. Reflections on applying constructive alignment with formative feedback for teaching introductory programming and software architecture. In Proceedings of the 38th International Conference on Software Engineering Companion. 336–345.

Catherine H Crouch and Eric Mazur. 2001. Peer instruction: Ten years of experience and results. American journal of physics 69, 9, 970–977.

Giuseppe De Ruvo, Ewan Tempero, Andrew Luxton-Reilly, Gerard B Rowe, and Nasser Giacaman. 2018. Understanding semantic style by analysing student code. In Proceedings of the 20th Australasian Computing Education Conference. 73–82.

Stephen H Edwards. 2021. Automated Feedback, the Next Generation: Designing Learning Experiences. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 610–611.

Margaret Ellis, Clifford A. Shaffer, and Stephen H. Edwards. 2019. Approaches for coordinating etextbooks, online programming practice, automated grading, and more into one course. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 126–132.

Leandro Galvão, David Fernandes, and Bruno Gadelha. 2016. Juiz online como ferramenta de apoio a uma metodologia de ensino híbrido em programação. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), Vol. 27. 140.

Guilherme Gama, Ricardo Caceffo, Renan Souza, Raysa Bennati, Tales Aparecida, Islene Garcia, and Rodolfo Azevedo. 2018. An Antipattern Documentation about Misconceptions related to an Introductory Programming Course in Python. Technical Report IC-18-19. Institute of Computing, University of Campinas. In English, 106 pages.

Philip Guo. 2014. Python Is Now the Most Popular Introductory Teaching Language at Top U.S. Universities. [link]. Online.

Jack Hollingsworth. 1960. Automatic Graders for Programming Classes. Commun. ACM 3, 10, 528–529.

Silas Hsu, Tiffany Wenting Li, Zhilin Zhang, Max Fowler, Craig Zilles, and Karrie Karahalios. 2021. Attitudes Surrounding an Imperfect AI Autograder. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 681, 15 pages.

Petri Ihantola and Andrew Petersen. 2019. Code complexity in introductory programming courses. In Proceedings of the 52nd Hawaii International Conference on System Sciences. 7662–7670.

Inside Higher Ed. 2018. Autograder issues upset students at Berkeley. [link]. (Acesso em 18/01/2021).

Hieke Keuning, Bastiaan Heeren, and Johan Jeuring. 2019. How teachers would help students to improve their code. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education. 119–125.

Hieke Keuning, Bastiaan Heeren, and Johan Jeuring. 2021. A tutoring system to learn code refactoring. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 562–568.

Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research methods in human-computer interaction. Morgan Kaufmann.

David Liu and Andrew Petersen. 2019. Static analyses in python programming courses. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 666–671.

Samiha Marwan, Nicholas Lytle, Joseph Jay Williams, and Thomas Price. 2019. The impact of adding textual explanations to next-step hints in a novice programming environment. In Proceedings of the 2019 ACM conference on innovation and technology in computer science education. 520–526.

Filipe D Pereira, Elaine HT Oliveira, David BF Oliveira, Alexandra I Cristea, Leandro SG Carvalho, Samuel C Fonseca, Armando Toda, and Seiji Isotani. 2020. Using learning analytics in the Amazonas: understanding students’ behaviour in introductory programming. British journal of educational technology 51, 4, 955–972.

Leo Porter, Cynthia Bailey Lee, and Beth Simon. 2013. Halving Fail Rates Using Peer Instruction: A Study of Four Computer Science Courses. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (Denver, Colorado, USA) (SIGCSE ’13). Association for Computing Machinery, New York, NY, USA, 177–182.

James Prather, Raymond Pettit, Kayla McMurry, Alani Peters, John Homer, and Maxine Cohen. 2018. Metacognitive difficulties faced by novice programmers in automated assessment tools. ICER 2018 - Proceedings of the 2018 ACM Conference on International Computing Education Research, 41–50.

Yizhou Qian and James Lehman. 2017. Students’ misconceptions and other difficulties in introductory programming: A literature review. ACM Transactions on Computing Education (TOCE) 18, 1, 1–24.

R.H. Sampieri, C.F. Collado, and P.B. Lucio. 2013. Metodologia de pesquisa. Penso.

Eryck Silva, Ricardo Caceffo, and Rodolfo Azevedo. 2021. Análise Estática de Código em Conjunto com Autograders. In Anais Estendidos do I Simpósio Brasileiro de Educação em Computação (On-line). SBC, Porto Alegre, RS, Brasil, 25–26.

Eryck Silva, Ricardo Caceffo, and Rodolfo Azevedo. 2022. Análise dos Tópicos Mais Abordados em Disciplinas de Introdução à Programação em Universidades Federais Brasileiras. In Anais do II Simpósio Brasileiro de Educação em Computação (Online). SBC, Porto Alegre, RS, Brasil, 29–39.

Eryck Silva, Ricardo Caceffo, and Rodolfo Azevedo. 2023. Misconceptions in Correct Code: rating the severity of undesirable programming behaviors in Python CS1 courses. Technical Report IC-23-01. Institute of Computing, University of Campinas. In English, 149 pages.

Beth Simon, Sarah Esper, Leo Porter, and Quintin Cutts. 2013. Student Experience in a Student-Centered Peer Instruction Classroom. In Proceedings of the Ninth Annual International ACM Conference on International Computing Education Research (San Diego, San California, USA) (ICER ’13). Association for Computing Machinery, New York, NY, USA, 129–136.

Beth Simon, Michael Kohanfars, Jeff Lee, Karen Tamayo, and Quintin Cutts. 2010. Experience Report: Peer Instruction in Introductory Computing (SIGCSE ’10). Association for Computing Machinery, New York, NY, USA, 341–345.

Beth Simon, Julian Parris, and Jaime Spacco. 2013. How We Teach Impacts Student Learning: Peer Instruction vs. Lecture in CS0. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (Denver, Colorado, USA) (SIGCSE ’13). Association for Computing Machinery, New York, NY, USA, 41–46.

Juha Sorva and Arto Vihavainen. 2016. Break Statement Considered. ACM Inroads 7, 3, 36–41.

Leo C. Ureel and Charles Wallace. 2019. Automated critique of early programming antipatterns. SIGCSE 2019 - Proceedings of the 50th ACM Technical Symposium on Computer Science Education, 738–744.

Kurt VanLehn. 2006. The behavior of tutoring systems. International journal of artificial intelligence in education 16, 3, 227–265.

Brian A Wichmann, AA Canning, DL Clutterbuck, LA Winsborrow, NJ Ward, and D William R Marsh. 1995. Industrial perspective on static analysis. Software Engineering Journal 10, 2, 69–75.