Evidence on the use of ChatGPT in teaching software modeling: a controlled experiment

• Samir B. Murad UFGD
• Fabricio F. S. Lemos UFGD
• Silvana M. Melo UFGD
• Leo Natan Paschoal PUCPR
• Jorge M. Prates UEMS

DOI: https://doi.org/10.5753/wei.2025.9261

Abstract

Chatbots LLMs have been explored across various domains and for different purposes, including as mechanisms for supporting education. When utilized as educational resources, it is essential to understand the effects of their use. This study describes a controlled experiment that analyzed the effects of using ChatGPT 3.5 in supporting software modeling education, specifically in the construction of UML diagrams. The experiment was designed to evaluate the effectiveness and efficiency of students in creating use case, class, and activity diagrams while also assessing the learning gains facilitated by using this resource. The results indicated that students who utilized ChatGPT demonstrated, on average, greater effectiveness and efficiency in producing the models. Furthermore, these students exhibited superior learning gains compared to those who engaged in modeling without the support of ChatGPT.

References

Abukhalaf, S., Hamdaqa, M., and Khomh, F. (2023). On codex prompt engineering for ocl generation: An empirical study. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pages 148–157.

Baidoo-Anu, D. and Ansah, L. O. (2023). Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning. Journal of AI, 7(1):52–62.

Basili, V. R., Caldiera, G., and Rombach, D. H. (1994). The Goal Question Metric Approach, volume I. John Wiley & Sons.

Cámara, J., Troya, J., Burgueño, L., and Vallecillo, A. (2023). On the assessment of generative ai in modeling tasks: an experience report with chatgpt and uml. Softw. Syst. Model., 22(3):781–793.

Chaaben, M. B., Burgueño, L., and Sahraoui, H. (2023). Towards using few-shot prompt learning for automating model completion. In Proceedings of the 45th International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER ’23, page 7–12. IEEE Press.

Choudhuri, R., Liu, D., Steinmacher, I., Gerosa, M., and Sarma, A. (2024). How far are we? the triumphs and trials of generative ai in learning software engineering. In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE), ICSE ’24, pages 2270–2282, Los Alamitos, CA, USA. IEEE Computer Society.

Eysenbach, G. et al. (2023). The role of chatgpt, generative language models, and artificial intelligence in medical education: a conversation with chatgpt and a call for papers. JMIR medical education, 9(1):e46885.

Golgiyaz, S. (2023). Chatgpt in computer software education. ICHEAS, 4th International Conference On Health, Engineering And Applied Sciences., pages 115–126.

Jalil, S., Rafi, S., LaToza, T. D., Moran, K., and Lam, W. (2023). Chatgpt and software testing education: Promises perils. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 4130–4137.

Khojah, R., Mohamad, M., Leitner, P., and de Oliveira Neto, F. G. (2024). Beyond code generation: An observational study of chatgpt usage in software engineering practice. Proc. ACM Softw. Eng., 1(FSE).

Kosar, T., Ostojić, D., Liu, Y. D., and Mernik, M. (2024). Computer science education in chatgpt era: Experiences from an experiment in a programming course for novice programmers. Mathematics, 12(5).

Li, Y., Xu, J., Zhu, Y., Liu, H., and Liu, P. (2023). The impact of chatgpt on software engineering education: A quick peek. In 2023 10th International Conference on Dependable Systems and Their Applications (DSA), pages 595–596.

Paschoal, L. N. (2024). Um framework para o planejamento de experimentos controlados na pesquisa de chatbots educacionais. Tese de doutorado, Instituto de Ciências Matemáticas e de Computação, University of São Paulo, São Carlos.

Rodriguez-Echeverría, R., Gutiérrez, J. D., Conejero, J. M., and Prieto, A. E. (2024). Analysis of chatgpt performance in computer engineering exams. IEEE Revista Ibero-americana de Tecnologias del Aprendizaje, 19:71–80.

Ronanki, K., Berger, C., and Horkoff, J. (2023). Investigating chatgpt’s potential to assist in requirements elicitation processes. In 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pages 354–361.

Ronanki, K., Cabrero-Daniel, B., Horkoff, J., and Berger, C. (2024). Requirements Engineering Using Generative AI: Prompts and Prompting Patterns, pages 109–127. Springer Nature Switzerland, Cham.

Russo, D. (2024). Navigating the complexity of generative ai adoption in software engineering. ACM Trans. Softw. Eng. Methodol., 33(5).

Sami, M. A., Rasheed, Z., Waseem, M., Zhang, Z., Herda, T., and Abrahamsson, P. (2024). Prioritizing software requirements using large language models. CoRR, abs/2405.01564.

Schäfer, M., Nadi, S., Eghbali, A., and Tip, F. (2024). An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1):85–105.

Wohlin, C., Runeson, P., Hst, M., Ohlsson, M. C., Regnell, B., and Wessln, A. (2012). Experimentation in Software Engineering. Springer Publishing Company, Incorporated.

Xue, Y., Chen, H., Bai, G. R., Tairas, R., and Huang, Y. (2024). Does chatgpt help with introductory programming?an experiment of students using chatgpt in cs1. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training, ICSE-SEET ’24, page 331–341, New York, NY, USA. Association for Computing Machinery.