LLMAgentGrader: Sistema Multiagente com DeepSeek para Correção Automática Aprimorada de Respostas Curtas
Resumo
A Correção Automática de Respostas Curtas (ASAG) é crucial para escalar a avaliação educacional, mas apresenta desafios em precisão. Modelos de Linguagem de Grande Porte (LLMs) como o GPT-4 demonstraram potencial, mas abordagens com LLMs únicos podem ter limitações. Este artigo apresenta o LLMAgentGrader, um sistema multiagente utilizando DeepSeek-Chat, projetado para ASAG. O sistema emprega agentes especializados para: (i) gerar respostas de referência (inclusive com busca na web), (ii) extrair conceitos principais das respostas, e (iii) sugerir notas com base em comparação semântica e completude, utilizando uma estratégia de few-shot learning dinâmico com um histórico de correções anteriores amostradas aleatoriamente. Esse estudo valida o LLMAgentGrader nos datasets TEXAS (inglês) e ASAG 2018 (português), comparando-o com resultados de um sistema GPT-4 e modelos tradicionais de Aprendizagem de Máquina. Os resultados demonstram que a abordagem multiagente com DeepSeek-Chat alcança um desempenho significativamente superior em termos de MAE e RMSE, superando não apenas o GPT-4, mas também alguns modelos tradicionais em cenários específicos.
Referências
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
Agirre, E., Cer, D., Diab, M., and Gonzalez-Agirre, A. (2012). Semeval-2012 task 6: A pilot on semantic textual similarity.* sem 2012: The first joint conference on lexical and computational semantics—. In Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), Montréal, QC, Canada, pages 7–8.
Almeida, J. A. O. d. S. and Moura, R. S. (2024). Investigação de métodos de similaridade textual no contexto da avaliação automática de questões discursivas. In Escola Regional de Computação do Ceará, Maranhão e Piauí(ERCEMAPI), pages 110–118. SBC.
Bitencourt, B. M., Severo, M. B., and Gallon, S. (2013). Avaliação da aprendizagem no ensino superior: desafios e potencialidades na educação a distância. Revista eletrônica de educação, 7(2):211–226.
Black, P., Harrison, C., and Lee, C. (2003). Assessment for learning: Putting it into practice. McGraw-Hill Education (UK).
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
Burrows, S., Gurevych, I., and Stein, B. (2015). The eras and trends of automatic short answer grading. International journal of artificial intelligence in education, 25:60–117.
Camus, L. and Filighera, A. (2020). Investigating transformers for automatic short answer grading. In Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part II 21, pages 43–48. Springer.
Condor, A., Litster, M., and Pardos, Z. (2021a). Automatic short answer grading with sbert on out-of-sample questions. International Educational Data Mining Society.
Condor, A. J., Litster, M., and Pardos, Z. A. (2021b). Automatic short answer grading with SBERT on out-of-sample questions. In Proceedings of the 14th International Conference on Educational Data Mining, EDM 2021, Paris, France, June 29 - July 2, 2021, pages 748–752. International Educational Data Mining Society (IEDMS).
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186.
Ferreira Mello, R., Pereira Junior, C., Rodrigues, L., Pereira, F. D., Cabral, L., Costa, N., Ramalho, G., and Gasevic, D. (2025). Automatic short answer grading in the llm era: Does gpt-4 with prompt engineering beat traditional models? In Proceedings of the 15th International Learning Analytics and Knowledge Conference, pages 93–103.
Fırat, M. and Kuleli, S. (2023). What if gpt4 became autonomous: The auto-gpt project and use cases. Journal of Emerging Computer Technologies, 3(1):1–6.
Galhardi, L., de Souza, R. C. T., and Brancher, J. (2020). Automatic grading of portuguese short answers using a machine learning approach. In Simpósio Brasileiro de Sistemas de Informação (SBSI), pages 109–124. SBC.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. (2024). Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.
Mohler, M., Bunescu, R., and Mihalcea, R. (2011). Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 752–762.
Naismith, B., Mulcaire, P., and Burstein, J. (2023). Automated evaluation of written discourse coherence using gpt-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 394–403.
Nguyen, H. A., Stec, H., Hou, X., Di, S., and McLaren, B. M. (2023). Evaluating chatgpt’s decimal skills and feedback generation in a digital learning game. In European conference on technology enhanced learning, pages 278–293. Springer.
Park, J. S., O’Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., Bernstein, M. S., et al. (2023). Generative agents: Interactive simulacra of human behavior. arxiv. Org (2023, April 7) [link] v2.
Sirotheau, S., Santos, J., Favero, E., and de Freitas, S. N. (2019). Avaliação automática de respostas discursivas curtas baseado em três dimensões linguísticas. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 30, page 1551.
Sung, C., Dhamecha, T. I., and Mukhi, N. (2019). Improving short answer grading using transformer-based pre-training. In Artificial Intelligence in Education: 20th International Conference, AIED 2019, Chicago, IL, USA, June 25-29, 2019, Proceedings, Part I 20, pages 469–481. Springer.
Süzen, N., Gorban, A. N., Levesley, J., and Mirkes, E. M. (2020). Automatic short answer grading and feedback using text mining methods. Procedia computer science, 169:726–743.
Agirre, E., Cer, D., Diab, M., and Gonzalez-Agirre, A. (2012). Semeval-2012 task 6: A pilot on semantic textual similarity.* sem 2012: The first joint conference on lexical and computational semantics—. In Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), Montréal, QC, Canada, pages 7–8.
Almeida, J. A. O. d. S. and Moura, R. S. (2024). Investigação de métodos de similaridade textual no contexto da avaliação automática de questões discursivas. In Escola Regional de Computação do Ceará, Maranhão e Piauí(ERCEMAPI), pages 110–118. SBC.
Bitencourt, B. M., Severo, M. B., and Gallon, S. (2013). Avaliação da aprendizagem no ensino superior: desafios e potencialidades na educação a distância. Revista eletrônica de educação, 7(2):211–226.
Black, P., Harrison, C., and Lee, C. (2003). Assessment for learning: Putting it into practice. McGraw-Hill Education (UK).
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
Burrows, S., Gurevych, I., and Stein, B. (2015). The eras and trends of automatic short answer grading. International journal of artificial intelligence in education, 25:60–117.
Camus, L. and Filighera, A. (2020). Investigating transformers for automatic short answer grading. In Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part II 21, pages 43–48. Springer.
Condor, A., Litster, M., and Pardos, Z. (2021a). Automatic short answer grading with sbert on out-of-sample questions. International Educational Data Mining Society.
Condor, A. J., Litster, M., and Pardos, Z. A. (2021b). Automatic short answer grading with SBERT on out-of-sample questions. In Proceedings of the 14th International Conference on Educational Data Mining, EDM 2021, Paris, France, June 29 - July 2, 2021, pages 748–752. International Educational Data Mining Society (IEDMS).
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186.
Ferreira Mello, R., Pereira Junior, C., Rodrigues, L., Pereira, F. D., Cabral, L., Costa, N., Ramalho, G., and Gasevic, D. (2025). Automatic short answer grading in the llm era: Does gpt-4 with prompt engineering beat traditional models? In Proceedings of the 15th International Learning Analytics and Knowledge Conference, pages 93–103.
Fırat, M. and Kuleli, S. (2023). What if gpt4 became autonomous: The auto-gpt project and use cases. Journal of Emerging Computer Technologies, 3(1):1–6.
Galhardi, L., de Souza, R. C. T., and Brancher, J. (2020). Automatic grading of portuguese short answers using a machine learning approach. In Simpósio Brasileiro de Sistemas de Informação (SBSI), pages 109–124. SBC.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. (2024). Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437.
Mohler, M., Bunescu, R., and Mihalcea, R. (2011). Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 752–762.
Naismith, B., Mulcaire, P., and Burstein, J. (2023). Automated evaluation of written discourse coherence using gpt-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 394–403.
Nguyen, H. A., Stec, H., Hou, X., Di, S., and McLaren, B. M. (2023). Evaluating chatgpt’s decimal skills and feedback generation in a digital learning game. In European conference on technology enhanced learning, pages 278–293. Springer.
Park, J. S., O’Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., Bernstein, M. S., et al. (2023). Generative agents: Interactive simulacra of human behavior. arxiv. Org (2023, April 7) [link] v2.
Sirotheau, S., Santos, J., Favero, E., and de Freitas, S. N. (2019). Avaliação automática de respostas discursivas curtas baseado em três dimensões linguísticas. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 30, page 1551.
Sung, C., Dhamecha, T. I., and Mukhi, N. (2019). Improving short answer grading using transformer-based pre-training. In Artificial Intelligence in Education: 20th International Conference, AIED 2019, Chicago, IL, USA, June 25-29, 2019, Proceedings, Part I 20, pages 469–481. Springer.
Süzen, N., Gorban, A. N., Levesley, J., and Mirkes, E. M. (2020). Automatic short answer grading and feedback using text mining methods. Procedia computer science, 169:726–743.
Publicado
24/11/2025
Como Citar
RODRIGUES NETO, José; ALVES, Gabriel; MELLO, Rafael Ferreira.
LLMAgentGrader: Sistema Multiagente com DeepSeek para Correção Automática Aprimorada de Respostas Curtas. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 36. , 2025, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 617-628.
DOI: https://doi.org/10.5753/sbie.2025.12550.
