GPT AI in Computer Science Education: A Systematic Mapping Study

Resumo


With the advent of GPT-AI, new possibilities in education emerged. However, it is challenging to determine how and when to apply these new technologies and understand their actual impact on teaching and learning. This study conducts a systematic mapping to gather, include, and classify scientific papers that investigated the subject of generative AI in CS education. 31 relevant studies that conducted empirical evaluations of the application of GPT-AI tools in CS education were collected. Our findings highlight challenges regarding plagiarism, learning perception, and AI capability. The main contribution of this study is to present research opportunities and provide a background for future studies that address the application of GPT-AI in CS education.

Referências

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Adiguzel, T., Kaya, M. H., and Cansu, F. K. (2023). Revolutionizing education with ai: Exploring the transformative potential of chatgpt. Contemporary Educational Technology, 15(3):ep429.

Afjal, M. (2023). Chatgpt and the ai revolution: a comprehensive investigation of its multidimensional impact and potential. Library Hi Tech, ahead-of-print(ahead-of-print).

Albonico, M. and Varela, P. J. (2023). A report on the use of chatgpt in software engineering and systems analysis courses. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering, pages 303–311.

Ansari, A. N., Ahmad, S., and Bhutta, S. M. (2023). Mapping the global evidence around the use of chatgpt in higher education: A systematic scoping review. Education and Information Technologies, pages 1–41.

Aruleba, K., Sanusi, I. T., Obaido, G., and Ogbuokiri, B. (2023). Integrating chatgpt in a computer science course: Students perceptions and suggestions. arXiv preprint arXiv:2402.01640.

Baber, H., Nair, K., Gupta, R., and Gurjar, K. (2024). The beginning of chatgpt–a systematic and bibliometric review of the literature. Information and Learning Sciences, 125(7/8):587–614.

Baker, J. (2000). The “classroom flip”. using web course management tools to become the guide on the side. In 11th International Conference on College Teaching and Learning, Jacksonville, FL.

BBC News (2023). Italy temporarily bans chatgpt over privacy concerns. [link]. Accessed: 2024-06-01.

Budhiraja, R., Joshi, I., Challa, J. S., Akolekar, H. D., and Kumar, D. (2024). “it’s not like jarvis, but it’s pretty close!”-examining chatgpt’s usage among undergraduate students in computer science. In Proceedings of the 26th Australasian Computing Education Conference, pages 124–133.

Caldarini, G., Jaf, S., and McGarry, K. (2022). A literature survey of recent advances in chatbots. Information, 13(1):41.

Carbonell, J. R. (1970). Ai in cai: An artificial-intelligence approach to computer-assisted instruction. IEEE transactions on man-machine systems, 11(4):190–202.

Dai, W., Lin, J., Jin, H., Li, T., Tsai, Y.-S., Gašević, D., and Chen, G. (2023). Can large language models provide feedback to students? a case study on chatgpt. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), pages 323–325. IEEE.

Dempere, J., Modugu, K., Hesham, A., and Ramasamy, L. K. (2023). The impact of chatgpt on higher education. Frontiers in Education, 8.

Denny, P., Kumar, V., and Giacaman, N. (2023). Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2023), pages 1136–1142, New York, NY, USA. Association for Computing Machinery.

Farhi, F., Jeljeli, R., Aburezeq, I., Dweikat, F. F., Al-shami, S. A., and Slamene, R. (2023). Analyzing the students’ views, concerns, and perceived ethics about chat gpt usage. Computers and Education: Artificial Intelligence, 5:100180.

Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., and Prather, J. (2022). The robots are coming: Exploring the implications of openai codex on introductory programming. In Proceedings of the 24th Australasian Computing Education Conference, pages 10–19.

FlexOS (2024). Generative ai top 150: The world’s most used ai tools. Accessed: 2024-06-02.

French, F., Levi, D., Maczo, C., Simonaityte, A., Triantafyllidis, S., and Varda, G. (2023). Creative use of openai in education: case studies from game development. Multimodal Technologies and Interaction, 7(8):81.

Gehringer, E. and Peddycord III, B. (2013). The inverted-lecture model: A case study in computer architecture. pages 489–494.

Geraldi, R. T., Reinehr, S., and Malucelli, A. (2020). Software product line applied to the internet of things: A systematic literature review. Information and Software Technology, 124:106293.

Ghassemi, M., Birhane, A., Bilal, M., Kankaria, S., Malone, C., Mollick, E., and Tustumi, F. (2023). Chatgpt one year on: who is using it, how and why? Nature, 624(7990):39– 41.

Hammad, R. and Bahja, M. (2023). Opportunities and challenges in educational chatbots. Trends, Applications, and Challenges of Chatbot Technology, pages 119–136.

Hanifi, K., Cetin, O., and Yilmaz, C. (2023). On chatgpt: Perspectives from software engineering students. In 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS), pages 196–205. IEEE.

Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., and Sorva, J. (2023). Exploring the responses of large language models to beginner programmers’ help requests. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1, pages 93–105.

Hidalgo, C. G., Bucheli-Guerrero, V. A., and Ordóñez-Eraso, H. A. (2023). Artificial intelligence and computer-supported collaborative learning in programming: A systematic mapping study. Tecnura, 27(75):175–206.

Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., and MacNeil, S. (2024). The effects of generative ai on computing students’ help-seeking preferences. In Proceedings of the 26th Australasian Computing Education Conference, pages 39–48.

Hwang, G.-J. and Chang, C.-Y. (2021). A review of opportunities and challenges of chatbots in education. Interactive Learning Environments, 31(7):4099–4112.

Jalil, S., Rafi, S., LaToza, T., Moran, K., and Lam, W. (2023). Chatgpt and software testing education: Promises & perils. arXiv.

Jonsson, M. and Tholander, J. (2022). Cracking the code: Co-coding with ai in creative programming education. In Proceedings of the 14th Conference on Creativity and Cognition, pages 5–14.

Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., and Grossman, T. (2023). Studying the effect of ai code generators on supporting novice learners in introductory programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–23.

Kiesler, N., Lohr, D., and Keuning, H. (2023). Exploring the potential of large language models to generate formative programming feedback. In 2023 IEEE Frontiers in Education Conference (FIE), pages 1–5. IEEE.

Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., and Linkman, S. (2009). Systematic literature reviews in software engineering - a systematic literature review. Information and Software Technology, 51(1):7–15.

Kitchenham, B. and Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Technical report EBSE-2007-01, EBSE Technical Report.

Kitchenham, B. A., Budgen, D., and Brereton, O. P. (2010). The value of mapping studies – a participant-observer case study.

Kokkoniemi, M. and Isomöttönen, V. (2023). A systematic mapping study on group work research in computing education projects. Journal of Systems and Software, 204:111795.

Lau, S. and Guo, P. (2023). From” ban it till we understand it” to” resistance is futile”: How university programming instructors plan to adapt as more students use ai code generation and explanation tools such as chatgpt and github copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1, pages 106–121.

Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., and Hellas, A. (2023). Comparing code explanations created by students and large language models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, pages 124–130.

Lemke, C., Kirchner, K., Anandarajah, L., and Herfurth, F. N. (2023). Exploring the student perspective: Assessing technology readiness and acceptance for adopting large language models in higher education. In 22nd European Conference on e-Learning: ECEL 2023. Academic Conferences and publishing limited.

Lin, M. P.-C., Chang, D., Hall, S., and Jhajj, G. (2024). Preliminary systematic review of open-source large language models in education. In International Conference on Intelligent Tutoring Systems, pages 68–77. Springer.

Lo, C. (2023). What is the impact of chatgpt on education? a rapid review of the literature. Education Sciences, 13(4):410.

Ma, B., Chen, L., and Konomi, S. (2024). Enhancing programming education with chatgpt: A case study on student perceptions and interactions in a python course. arXiv preprint arXiv:2403.15472.

Maher, M. L., Latulipe, C., Lipford, H., and Rorrer, A. (2015). Flipped classroom strategies for cs education. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education, SIGCSE ’15, page 218–223, New York, NY, USA. Association for Computing Machinery.

Maher, M. L., Tadimalla, S. Y., and Dhamani, D. (2023). An exploratory study on the impact of ai tools on the student experience in programming courses: an intersectional analysis approach. In 2023 IEEE Frontiers in Education Conference (FIE), pages 1–5. IEEE.

Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, J. (2024). Large language models: A survey. arXiv preprint arXiv:2402.06196.

Mosaiyebzadeh, F., Pouriyeh, S., Parizi, R., Dehbozorgi, N., Dorodchi, M., and Batista, D. (2023). Exploring the role of chatgpt in education: Applications and challenges. pages 84–89.

OpenAI (2022). Introducing chatgpt. [link]. Accessed: 2024-06-01.

Petersen, K., Vakkalanka, S., and Kuzniarz, L. (2015). Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology, 64:1–18.

Popovici, M.-D. (2023). Chatgpt in the classroom. exploring its potential and limitations in a functional programming course. International Journal of Human–Computer Interaction, pages 1–12.

Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023a). The robots are here: Navigating the generative ai revolution in computing education. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education, pages 108–159.

Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., and Santos, E. A. (2023b). “it’s weird that it knows what i want”: Usability and interactions with copilot for novice programmers. ACM Transactions on Computer-Human Interaction, 31(1):1–31.

Purnama, I., Edi, F., Agustin, R., Pranoto, N. W., et al. (2023). Gpt chat integration in project based learning in learning: a systematic literature review. Jurnal Penelitian Pendidikan IPA, 9(SpecialIssue):150–158.

Rahman, M. M. and Watanobe, Y. (2023). Chatgpt for education and research: Opportunities, threats, and strategies. Applied Sciences, 13(9):5783.

Rajala, J., Hukkanen, J., Hartikainen, M., and Niemelä, P. (2023). ”call me kiran” – chatgpt as a tutoring chatbot in a computer science course. In Proceedings of the 26th International Academic Mindtrek Conference, Mindtrek ’23, page 83–94, New York, NY, USA. Association for Computing Machinery.

Reiche, M. and Leidner, J. L. (2023). Bridging the programming skill gap with chatgpt: A machine learning project with business students. In European Conference on Artificial Intelligence, pages 439–446. Springer.

Rotman, D. (2023). How chatgpt will revolutionize the economy. we need to decide what that looks like. MIT Technology Review.

Sheese, B., Liffiton, M., Savelka, J., and Denny, P. (2024). Patterns of student help-seeking when using a large language model-powered programming assistant. In Proceedings of the 26th Australasian Computing Education Conference, pages 49–57.

Silva, C. A. G. d., Ramos, F. N., de Moraes, R. V., and Santos, E. L. d. (2024). Chatgpt: Challenges and benefits in software programming for higher education. Sustainability, 16(3):1245.

Sun, D., Boudouaia, A., Zhu, C., and Li, Y. (2024). Would chatgpt-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? an empirical study. International Journal of Educational Technology in Higher Education, 21(1):14.

Williamson, B. (2024). The social life of ai in education. International Journal of Artificial Intelligence in Education, 34(1):97–104.

Yan, L., Sha, L., Zhao, L., Li, Y., Martinez-Maldonado, R., Chen, G., Li, X., Jin, Y., and Gašević, D. (2024). Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Technology, 55(1):90–112.

Yilmaz, R. and Yilmaz, F. G. K. (2023a). Augmented intelligence in programming learning: Examining student views on the use of chatgpt for programming learning. Computers in Human Behavior: Artificial Humans, 1(2):100005.

Yilmaz, R. and Yilmaz, F. G. K. (2023b). The effect of generative artificial intelligence (ai)-based tool use on students’ computational thinking skills, programming self-efficacy and motivation. Computers and Education: Artificial Intelligence, 4:100147.

Zastudil, C., Rogalska, M., Kapp, C., Vaughn, J., and MacNeil, S. (2023). Generative ai in computing education: Perspectives of students and instructors. In 2023 IEEE Frontiers in Education Conference (FIE), pages 1–9. IEEE.

Zheng, Y. (2023). Chatgpt for teaching and learning: An experience from data science education. In Proceedings of the 24th Annual Conference on Information Technology Education, pages 66–72.
Publicado
04/11/2024
STRIK, Bruno H.; MENOLLI, André; BRANCHER, Jacques Duílio. GPT AI in Computer Science Education: A Systematic Mapping Study. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 35. , 2024, Rio de Janeiro/RJ. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2024 . p. 1543-1559. DOI: https://doi.org/10.5753/sbie.2024.242103.