Feedback Formativo Automatizado com LLMs: Desenvolvimento e Análise de um Sistema para Aprendizagem Progressiva em Programação
Resumo
Oferecer feedback formativo de qualidade a estudantes de programação é um dos principais desafios educacionais, sobretudo em ambientes com grande número de participantes. A partir desse contexto, este trabalho apresenta o desenvolvimento e a validação de um framework adaptativo baseado em grandes modelos de linguagem (LLMs) para avaliação automática de código e geração de feedback personalizado. A ferramenta emprega estratégias pedagógicas e técnicas avançadas de engenharia de prompt para promover mediação conceitual e apoio ao estudante. A análise de 300 tentativas reais evidenciou 74,7% de concordância com autograders tradicionais e 93,3% de feedbacks qualitativamente avaliados como coerentes, destacando potencial para apoiar a aprendizagem significativa e ampliar o alcance de práticas avaliativas inovadoras.Referências
Ala-Mutka, K. (2005). A survey of automated assessment approaches for programming assignments. In Computer Science Education, volume 15, page 83–102.
Alvim, I., Bittencourt, R., and Duran, R. (2024). Evasão nos cursos de graduação em computação no brasil. In Simpósio Brasileiro de Educação em Computação (EDUCOMP).
Ausubel, D. P. (1963). The psychology of meaningful verbal learning. Grune & Stratton.
Bassner, P., Frankford, E., and Krusche, S. (2024). Iris: An ai-driven virtual tutor for computer science education. In ITiCSE 2024: Proceedings of the 29th ACM Conference on Innovation and Technology in Computer Science Education, pages 1–7.
Black, P. and Wiliam, D. (2009). Developing the Theory of Formative Assessment, volume 21. Educational Assessment, Evaluation and Accountability.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, page 1877–1901.
Cheng, G., Santos, A., Koh, E., and Cheung, S. (2023). Codetailor: Llm-powered personalized parsons puzzles for engaging support while learning programming. In Proceedings of the 18th European Conference on Technology Enhanced Learning (EC-TEL), pages 143–156.
Ding, Y., Hu, H., Zhou, J., Chen, Q., Jiang, B., and He, L. (2024). Boosting large language models with socratic method for conversational mathematics teaching. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM 2024), pages 3730–3735.
Gill, S. S., Buyya, R., Yang, C., and Chan, C. S. (2024). Edge ai: A taxonomy, systematic review and future directions. Cluster Computing, 27(1):99–121.
Ihantola, P., Ahoniemi, T., Karavirta, V., and Seppälä, O. (2010). Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, page 86–93.
Jang, J., Eun, S., and Lee, H. (2024). The effects of prompt scaffolding on learning to write arguments with chatgpt. In Proceedings of the 17th International Conference of the Learning Sciences (ICLS 2024), pages 1502–1505.
Kasneci, E., Sessler, K., Völkel, T., and Kasneci, G. (2023). Chatgpt for good? on opportunities and challenges of large language models for education. Learning and Individual Differences, 103:102274.
Kazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., and Grossman, T. (2024). Codeaid: Evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Machinery.
Keuning, H., Jeuring, J., and Heeren, B. (2018). A systematic literature review of automated feedback generation for programming exercises. ACM Transactions on Computing Education, 19(1):1–43.
Krupp, L., Steinert, S., Kiefer-Emmanouilidis, M., Avila, K. E., Lukowicz, P., Kuhn, J., Küchemann, S., and Karolus, J. (2024). Challenges and opportunities of moderating usage of large language models in education. MUM ’24, page 249–254, New York, NY, USA. Association for Computing Machinery.
Lampou, R. (2023). The integration of artificial intelligence in education: Opportunities and challenges. Review of Artificial Intelligence in Education, 4.
Leite, A. and Blanco, S. A. (2020). Effects of human vs. automatic feedback on students’ understanding of ai concepts and programming style. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education, SIGCSE ’20, page 44–50, New York, NY, USA. Association for Computing Machinery.
Luckin, R., Holmes, W., Griffiths, M., and Forcier, L. (2016). Intelligence Unleashed: An Argument for AI in Education. Pearson.
Marvin, G., Hellen, N., Jjingo, D., and Nakatumba-Nabende, J. (2024). Prompt engineering in large language models. In Jacob, I. J., Piramuthu, S., and Falkowski-Gilski, P., editors, Data Intelligence and Cognitive Informatics, pages 387–402, Singapore. Springer Nature Singapore.
Medeiros, R. P., Ramalho, G. L., and Falcão, T. P. (2019). A systematic literature review on teaching and learning introductory programming in higher education. IEEE Transactions on Education, 62(2):77–90.
Messer, M., Brown, N. C. C., Kölling, M., and Shi, M. (2023). Automated grading and feedback tools for programming education: A systematic review. ACM Transactions on Computing Education, 24(1):1–34.
Nicol, D. J. and Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2):199–218.
Paiva, J. C., Leal, J. P., and Figueira, A. (2022). Automated assessment in computer science education: A state-of-the-art review. ACM Trans. Comput. Educ., 22(3).
Pereira, A. F. and Ferreira Mello, R. (2025). A systematic literature review on large language models applications in computer programming teaching evaluation process. IEEE Access, 13:113449–113460.
Porayska-Pomsta, K., Holmes, W., and Nemorin, S. (2024). The ethics of ai in education. In du Boulay, B., Mitrovic, A., and Yacef, K., editors, Handbook of Artificial Intelligence in Education, pages 571–604. Edward Elgar.
Pornprasit, C. and Tantithamthavorn, C. (2024). Fine-tuning and prompt engineering for large language models-based code review automation. Information and Software Technology, 175:107523.
Schwerter, J., Schindler, K., Hoffmann, K., and Sailer, M. (2022). E-learning with multiple-try-feedback: Can hints foster students’ achievement? Educational Technology Research and Development, 70:713–736.
Teusner, R., Hille, T., and Staubitz, T. (2018). Effects of automated interventions in programming assignments: evidence from a field experiment. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale, L@S ’18, New York, NY, USA. Association for Computing Machinery.
Vygotsky, L. S. (1978). Mind in Society: The Development of Higher Psychological Processes. Harvard University Press.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Piloto, L., Xia, F., Tillet, P., Le, Q. V., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
Yan, L., Sha, L., Zhao, L., Li, Y., Maldonado, R. M., Chen, G., Li, X., Jin, Y., and Gašević, D. (2024). Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Technology, 55(1):90–112.
Yu, Z., Xie, X., and Xu, H. (2023). Automated feedback generation in programming education: A review of recent advances. IEEE Transactions on Learning Technologies, 16(1):78–91.
Zawacki-Richter, O., Marín, V. I., Bond, M., and Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education, 16(39).
Zhou, M., Chen, F., and Lee, H. F. (2023). Ethics in the age of artificial intelligence in education: Promises and pitfalls. British Journal of Educational Technology, 54(1):10–29.
Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice, 41(2):64–70.
Alvim, I., Bittencourt, R., and Duran, R. (2024). Evasão nos cursos de graduação em computação no brasil. In Simpósio Brasileiro de Educação em Computação (EDUCOMP).
Ausubel, D. P. (1963). The psychology of meaningful verbal learning. Grune & Stratton.
Bassner, P., Frankford, E., and Krusche, S. (2024). Iris: An ai-driven virtual tutor for computer science education. In ITiCSE 2024: Proceedings of the 29th ACM Conference on Innovation and Technology in Computer Science Education, pages 1–7.
Black, P. and Wiliam, D. (2009). Developing the Theory of Formative Assessment, volume 21. Educational Assessment, Evaluation and Accountability.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, page 1877–1901.
Cheng, G., Santos, A., Koh, E., and Cheung, S. (2023). Codetailor: Llm-powered personalized parsons puzzles for engaging support while learning programming. In Proceedings of the 18th European Conference on Technology Enhanced Learning (EC-TEL), pages 143–156.
Ding, Y., Hu, H., Zhou, J., Chen, Q., Jiang, B., and He, L. (2024). Boosting large language models with socratic method for conversational mathematics teaching. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM 2024), pages 3730–3735.
Gill, S. S., Buyya, R., Yang, C., and Chan, C. S. (2024). Edge ai: A taxonomy, systematic review and future directions. Cluster Computing, 27(1):99–121.
Ihantola, P., Ahoniemi, T., Karavirta, V., and Seppälä, O. (2010). Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, page 86–93.
Jang, J., Eun, S., and Lee, H. (2024). The effects of prompt scaffolding on learning to write arguments with chatgpt. In Proceedings of the 17th International Conference of the Learning Sciences (ICLS 2024), pages 1502–1505.
Kasneci, E., Sessler, K., Völkel, T., and Kasneci, G. (2023). Chatgpt for good? on opportunities and challenges of large language models for education. Learning and Individual Differences, 103:102274.
Kazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., and Grossman, T. (2024). Codeaid: Evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Machinery.
Keuning, H., Jeuring, J., and Heeren, B. (2018). A systematic literature review of automated feedback generation for programming exercises. ACM Transactions on Computing Education, 19(1):1–43.
Krupp, L., Steinert, S., Kiefer-Emmanouilidis, M., Avila, K. E., Lukowicz, P., Kuhn, J., Küchemann, S., and Karolus, J. (2024). Challenges and opportunities of moderating usage of large language models in education. MUM ’24, page 249–254, New York, NY, USA. Association for Computing Machinery.
Lampou, R. (2023). The integration of artificial intelligence in education: Opportunities and challenges. Review of Artificial Intelligence in Education, 4.
Leite, A. and Blanco, S. A. (2020). Effects of human vs. automatic feedback on students’ understanding of ai concepts and programming style. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education, SIGCSE ’20, page 44–50, New York, NY, USA. Association for Computing Machinery.
Luckin, R., Holmes, W., Griffiths, M., and Forcier, L. (2016). Intelligence Unleashed: An Argument for AI in Education. Pearson.
Marvin, G., Hellen, N., Jjingo, D., and Nakatumba-Nabende, J. (2024). Prompt engineering in large language models. In Jacob, I. J., Piramuthu, S., and Falkowski-Gilski, P., editors, Data Intelligence and Cognitive Informatics, pages 387–402, Singapore. Springer Nature Singapore.
Medeiros, R. P., Ramalho, G. L., and Falcão, T. P. (2019). A systematic literature review on teaching and learning introductory programming in higher education. IEEE Transactions on Education, 62(2):77–90.
Messer, M., Brown, N. C. C., Kölling, M., and Shi, M. (2023). Automated grading and feedback tools for programming education: A systematic review. ACM Transactions on Computing Education, 24(1):1–34.
Nicol, D. J. and Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2):199–218.
Paiva, J. C., Leal, J. P., and Figueira, A. (2022). Automated assessment in computer science education: A state-of-the-art review. ACM Trans. Comput. Educ., 22(3).
Pereira, A. F. and Ferreira Mello, R. (2025). A systematic literature review on large language models applications in computer programming teaching evaluation process. IEEE Access, 13:113449–113460.
Porayska-Pomsta, K., Holmes, W., and Nemorin, S. (2024). The ethics of ai in education. In du Boulay, B., Mitrovic, A., and Yacef, K., editors, Handbook of Artificial Intelligence in Education, pages 571–604. Edward Elgar.
Pornprasit, C. and Tantithamthavorn, C. (2024). Fine-tuning and prompt engineering for large language models-based code review automation. Information and Software Technology, 175:107523.
Schwerter, J., Schindler, K., Hoffmann, K., and Sailer, M. (2022). E-learning with multiple-try-feedback: Can hints foster students’ achievement? Educational Technology Research and Development, 70:713–736.
Teusner, R., Hille, T., and Staubitz, T. (2018). Effects of automated interventions in programming assignments: evidence from a field experiment. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale, L@S ’18, New York, NY, USA. Association for Computing Machinery.
Vygotsky, L. S. (1978). Mind in Society: The Development of Higher Psychological Processes. Harvard University Press.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Piloto, L., Xia, F., Tillet, P., Le, Q. V., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
Yan, L., Sha, L., Zhao, L., Li, Y., Maldonado, R. M., Chen, G., Li, X., Jin, Y., and Gašević, D. (2024). Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Technology, 55(1):90–112.
Yu, Z., Xie, X., and Xu, H. (2023). Automated feedback generation in programming education: A review of recent advances. IEEE Transactions on Learning Technologies, 16(1):78–91.
Zawacki-Richter, O., Marín, V. I., Bond, M., and Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education, 16(39).
Zhou, M., Chen, F., and Lee, H. F. (2023). Ethics in the age of artificial intelligence in education: Promises and pitfalls. British Journal of Educational Technology, 54(1):10–29.
Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice, 41(2):64–70.
Publicado
24/11/2025
Como Citar
SILVA, Francisco Genivan; ARANHA, Eduardo Henrique da Silva.
Feedback Formativo Automatizado com LLMs: Desenvolvimento e Análise de um Sistema para Aprendizagem Progressiva em Programação. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 36. , 2025, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 1158-1172.
DOI: https://doi.org/10.5753/sbie.2025.12807.
