Evaluating resolutions of introductory programming exercises in the era of Generative AIs: A Case Study with ChatGPT

• William Simão de Deus University of São Paulo http://orcid.org/0000-0001-7540-3742
• Anderson da Silva Marcolino Federal University of Paraná https://orcid.org/0000-0002-4014-1882
• Gustavo Martins Nunes Avellar University of São Paulo https://orcid.org/0000-0002-3340-6766
• Katyeudo Karlos de Sousa Oliveira University of São Paulo https://orcid.org/0000-0002-4414-1599
• Ellen Francine Barbosa University of São Paulo https://orcid.org/0000-0003-3275-2293

DOI: https://doi.org/10.5753/sbie.2024.242283

Abstract

This paper describes a case study on how ChatGPT approaches and solves introductory computer programming problems, identifying elements that demonstrate whether the code was produced by generative Artificial Intelligence (AI) and not by a beginner in programming. To this end, 1.258 lines of code produced by ChatGPT were analyzed. Results provided patterns and practices that are not usually observed by beginners in programming. This study provides an analysis and suggestions considering the practical application for educators.

Keywords: Generative artificial intelligence, AI, introductory programming, exercise

References

Biderman, S. and Raff, E. (2022). Fooling moss detection with pretrained language models. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 2933–2943, New York, NY, USA. Association for Computing Machinery.

Biswas, S. (2023). Role of chatgpt in computer programming.: Chatgpt in computer programming. Mesopotamian Journal of Computer Science, 2023:8–16.

Bull, C. and Kharrufa, A. (2023). Generative ai assistants in software development education: A vision for integrating generative ai into educational practice, not instinctively defending against it. IEEE Software.

Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., and Reeves, B. N. (2024a). Prompt problems: A new programming exercise for the generative ai era. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, pages 296–302.

Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., and Sarsa, S. (2024b). Computing education in the era of generative ai. Commun. ACM, 67(2):56–67.

Euchner, J. (2023). Generative ai. Research-Technology Management, 66(3):71–74.

Feuerriegel, S., Hartmann, J., Janiesch, C., and Zschech, P. (2024). Generative ai. Business & Information Systems Engineering, 66(1):111–126.

Filho, L. P., Souza, T., and Paula, L. (2023). Análise das Respostas do ChatGPT em Relação ao Conteúdo de Programação para Iniciantes. In Anais do XXXIV Simpósio Brasileiro de Informática na Educação, pages 1738–1748, Porto Alegre, RS, Brasil. SBC.

Imran, M. and Almusharraf, N. (2023). Analyzing the role of chatgpt as a writing assistant at higher education level: A systematic review of the literature. Contemporary Educational Technology, 15(4):ep464.

Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., and Lo, D. (2024). Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Trans. Softw. Eng. Methodol. Just Accepted.

Medeiros, R. P., Ramalho, G. L., and Falcão, T. P. (2018). A systematic literature review on teaching and learning introductory programming in higher education. IEEE Transactions on Education, 62(2):77–90.

Oliveira, K. K. d. S., Marcolino, A. d. S., Deus, W. S. d., Falcão, T. P. d. R., and Barbosa, E. F. (2023). Pensamento computacional na programação introdutória e habilidades do século xxi: um mapeamento sistemático da literatura. Revista Novas Tecnologias na Educação-RENOTE, 21(2):519–531.

Qian, Y. and Lehman, J. (2017). Students’ misconceptions and other difficulties in introductory programming: A literature review. ACM Transactions on Computing Education (TOCE), 18(1):1–24.

Sharples, M. (2023). Towards social generative ai for education: theory, practices and ethics. Learning: Research and Practice, 9(2):159–167.

Stack Overflow, D. S. (2023). 2023 developer survey: Ai search tools.

Taecharungroj, V. (2023). “what can chatgpt do?” analyzing early reactions to the innovative ai chatbot on twitter. Big Data and Cognitive Computing, 7(1):35.

Tew, A. E. and Guzdial, M. (2010). Developing a validated assessment of fundamental cs1 concepts. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education, page 97–101, Milwaukee, Wisconsin, USA. Proceedings [...]. New York, NY, USA. Association for Computing Machinery.

Virvou, M., Tsihrintzis, G. A., Sotiropouloss, D. N., Chrysafiadi, K., Sakkopoulos, E., and Tsichrintzi, E.-A. (2023). Chatgpt in artificial intelligence-empowered e-learning for cultural heritage: The case of lyrics and poems. In 2023 14th International Conference on Information, Intelligence, Systems & Applications (IISA), pages 1–9. IEEE.

Yin, R. K. (2015). Estudo de Caso: Planejamento e métodos. Bookman editora