Exploring ChatGPT for automatic generation of Computer Programming practice questions

• Fernando Euzebio Zimerman Federal University of Ouro Preto http://orcid.org/0009-0003-7932-6950
• Fernando Henrique Oliveira Duarte Federal University of Ouro Preto https://orcid.org/0009-0003-0228-0712
• Pedro Henrique Lopes Silva Federal University of Ouro Preto https://orcid.org/0000-0002-5525-6121
• Reinaldo Silva Fortes Federal University of Ouro Preto https://orcid.org/0000-0002-2032-3291

DOI: https://doi.org/10.5753/sbie.2024.242440

Abstract

The increasing demand for coding skills in today's digital age has highlighted the importance of effective educational materials that can help individuals learn how to code. Developing practical exercises for an introduction to programming can be challenging. Creating exercise datasets manually is a heavy task, but using Natural Language Processing (NLP) tools for text generation can automate this process and make it more scalable. This paper discusses the experiences in developing strategies to assess the feasibility of automatically creating introductory programming exercises by leveraging standard information and prompt engineering with NLP models.

Keywords: ChatGPT, Large Language Models, Automatic question generation, Prompt engineering

References

Cambria, E. and White, B. (2014). Jumping nlp curves: A review of natural language processing research [review article]. IEEE Computational Intelligence Magazine, 9(2):48–57.

Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., and Prather, J. (2022). The robots are coming: Exploring the implications of openai codex on introductory programming. In Proceedings of the 24th Australasian Computing Education Conference, ACE ’22, page 10–19, New York, NY, USA. Association for Computing Machinery.

Gomes, A. and Mendes, A. (2007). Learning to program - difficulties and solutions. pages 283–287

Kurdi, G., Leo, J., Parsia, B., Sattler, U., and Al-Emari, S. (2020). A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30:121–204.

OpenAI (2024a). ChatGPT. Disponível em: [link]. Acessado em Abril de 2024.

OpenAI (2024b). Openai codex. Disponível em: [link]. Acessando em Abril de 2024.

Sarsa, S., Denny, P., Hellas, A., and Leinonen, J. (2022). Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1, ICER ’22, page 27–43, New York, NY, USA. Association for Computing Machinery.

Steve Mollman (2022). Chatgpt gained 1 million users in under a week. here’s why the ai chatbot is primed to disrupt search as we know it. Disponível em: [link]. Acesso em: 27 de junho 2023.

Thorell M., Fridorff-Jens P.K., L. P. e. a. (2015). Transforming students into digital academics: a challenge at both the individual and the institutional level. BMC Med Educ 15, 48 (2015). DOI: 10.1186/s12909-015-0330-5.