Um chatbot treinado para entender intenções de estudantes em diálogos sobre teste de software
Resumo
Este artigo apresenta a evolução do TOB-STT, um chatbot desenvolvido para apoiar o ensino de teste de software. O chatbot auxilia os estudantes a sanar dúvidas, exemplificar conceitos e demonstrar a aplicação de critérios de teste para a geração de casos de teste. A versão inicial, baseada em casamento de padrões, embora promissora, revelou limitações na precisão das respostas e na compreensão das intenções dos estudantes. Com o propósito de aprimorar a qualidade das interações e a eficácia do suporte educacional, sua arquitetura foi reformulada com técnicas de processamento de linguagem natural e entendimento de linguagem natural, resultando em uma nova versão, denominada TOB-STT 2.0. Para avaliá-la, foi conduzido um estudo experimental entre as duas versões. Os resultados demonstraram que a nova versão foi consideravelmente mais eficaz em responder corretamente às intenções dos estudantes. Essas evidências indicam que a evolução do chatbot foi bem-sucedida em aprimorar sua capacidade de compreensão e, consequentemente, a qualidade do suporte educacional oferecido.Referências
AbuShawar, B. and Atwell, E. (2016). Usefulness, localizability, humanness, and language-benefit: additional evaluation criteria for natural language dialogue systems. International Journal of Speech Technology, 19:373–383.
Adamopoulou, E. and Moussiades, L. (2020). An overview of chatbot technology. In Artificial Intelligence Applications and Innovations: 16th IFIP WG 12.5 International Conference, AIAI 2020, Neos Marmaras, Greece, June 5–7, 2020, Proceedings, Part II 16, pages 373–383. Springer.
Attigeri, G., Agrawal, A., and Kolekar, S. V. (2024). Advanced nlp models for technical university information chatbots: Development and comparative analysis. IEEE Access, 12:29633–29647.
Basili, V. R., Caldiera, G., and Rombach, D. H. (1994). The Goal Question Metric Approach, volume I. John Wiley & Sons.
Groothuijsen, S., van den Beemt, A., Remmers, J. C., and van Meeuwen, L. W. (2024). Ai chatbots in programming education: students’ use in a scientific computing course and consequences for learning. Computers and Education: Artificial Intelligence, 7:100290.
Haldar, S., Pierce, M., and Fernando Capretz, L. (2025). Exploring the integration of generative ai tools in software testing education: A case study on chatgpt and copilot for preparatory testing artifacts in postgraduate learning. IEEE Access, 13:46070–46090.
Herpich, F., Nunes, F. B., Voss, G. B., and Medina, R. D. (2020). Three-dimensional virtual environment and npc: A perspective about intelligent agents ubiquitous. In Natural Language Processing: Concepts, Methodologies, Tools, and Applications, pages 912–938. IGI Global.
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., and Lam, W. (2023). Chatgpt and software testing education: Promises & perils. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 4130–4137.
Kuhail, M. A., Alturki, N., Alramlawi, S., and Alhejori, K. (2023). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1):973–1018.
Lee, D. and Yeo, S. (2022). Developing an ai-based chatbot for practicing responsive teaching in mathematics. Computers & Education, 191:104646.
Leifheit, B. R., Basso, F. P., and Silva, W. (2023). Characterizing toolkits for platform independent chatbot development. In Proceedings of the XIX Brazilian Symposium on Information Systems, pages 28–36.
Mohamed, A. M. (2024). Exploring the potential of an ai-based chatbot (chatgpt) in enhancing english as a foreign language (efl) teaching: perceptions of efl faculty members. Education and Information Technologies, 29(3):3195–3217.
Nee, C. K., Rahman, M. H. A., Yahaya, N., Ibrahim, N. H., Razak, R. A., and Sugino, C. (2023). Exploring the trend and potential distribution of chatbot in education: A systematic review. International Journal of Information and Education Technology, 13(3):516–525.
Paschoal, L. N. (2024). Um framework para o planejamento de experimentos controlados na pesquisa de chatbots educacionais. Tese de doutorado, Instituto de Ciências Matemáticas e de Computação, University of São Paulo, São Carlos.
Paschoal, L. N., de Oliveira, M. M., and Chicon, P. M. M. (2018). A chatterbot sensitive to student’s context to help on software engineering education. In Latin American Computer Conference, pages 839–848.
Paschoal, L. N., Melo, S. M., Neves, V. d. O., Conte, T. U., and de SOUZA, S. d. R. S. (2023). An experimental study on a conversational agent in software testing lessons. Informatics in Education, 22(1):99–120.
Paschoal, L. N., Turci, L. F., Conte, T. U., and Souza, S. R. (2019). Towards a conversational agent to support the software testing education. In Brazilian Symposium on Software Engineering, pages 57–66.
Pérez-Soler, S., Guerra, E., and De Lara, J. (2020). Model-driven chatbot development. In International Conference on Conceptual Modeling, pages 207–222. Springer.
Pérez-Soler, S., Juárez-Puerta, S., Guerra, E., and de Lara, J. (2021). Choosing a chatbot development tool. IEEE Software, 38(4):94–103.
Roca, M. D. L., Chan, M. M., Garcia-Cabot, A., Garcia-Lopez, E., and Amado-Salvatierra, H. (2024). The impact of a chatbot working as an assistant in a course for supporting student learning and engagement. Computer Applications in Engineering Education, 32(5):e22750.
Wohlin, C., Runeson, P., Hst, M., Ohlsson, M. C., Regnell, B., and Wessln, A. (2012). Experimentation in Software Engineering. Springer Publishing Company, Incorporated.
Adamopoulou, E. and Moussiades, L. (2020). An overview of chatbot technology. In Artificial Intelligence Applications and Innovations: 16th IFIP WG 12.5 International Conference, AIAI 2020, Neos Marmaras, Greece, June 5–7, 2020, Proceedings, Part II 16, pages 373–383. Springer.
Attigeri, G., Agrawal, A., and Kolekar, S. V. (2024). Advanced nlp models for technical university information chatbots: Development and comparative analysis. IEEE Access, 12:29633–29647.
Basili, V. R., Caldiera, G., and Rombach, D. H. (1994). The Goal Question Metric Approach, volume I. John Wiley & Sons.
Groothuijsen, S., van den Beemt, A., Remmers, J. C., and van Meeuwen, L. W. (2024). Ai chatbots in programming education: students’ use in a scientific computing course and consequences for learning. Computers and Education: Artificial Intelligence, 7:100290.
Haldar, S., Pierce, M., and Fernando Capretz, L. (2025). Exploring the integration of generative ai tools in software testing education: A case study on chatgpt and copilot for preparatory testing artifacts in postgraduate learning. IEEE Access, 13:46070–46090.
Herpich, F., Nunes, F. B., Voss, G. B., and Medina, R. D. (2020). Three-dimensional virtual environment and npc: A perspective about intelligent agents ubiquitous. In Natural Language Processing: Concepts, Methodologies, Tools, and Applications, pages 912–938. IGI Global.
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., and Lam, W. (2023). Chatgpt and software testing education: Promises & perils. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 4130–4137.
Kuhail, M. A., Alturki, N., Alramlawi, S., and Alhejori, K. (2023). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1):973–1018.
Lee, D. and Yeo, S. (2022). Developing an ai-based chatbot for practicing responsive teaching in mathematics. Computers & Education, 191:104646.
Leifheit, B. R., Basso, F. P., and Silva, W. (2023). Characterizing toolkits for platform independent chatbot development. In Proceedings of the XIX Brazilian Symposium on Information Systems, pages 28–36.
Mohamed, A. M. (2024). Exploring the potential of an ai-based chatbot (chatgpt) in enhancing english as a foreign language (efl) teaching: perceptions of efl faculty members. Education and Information Technologies, 29(3):3195–3217.
Nee, C. K., Rahman, M. H. A., Yahaya, N., Ibrahim, N. H., Razak, R. A., and Sugino, C. (2023). Exploring the trend and potential distribution of chatbot in education: A systematic review. International Journal of Information and Education Technology, 13(3):516–525.
Paschoal, L. N. (2024). Um framework para o planejamento de experimentos controlados na pesquisa de chatbots educacionais. Tese de doutorado, Instituto de Ciências Matemáticas e de Computação, University of São Paulo, São Carlos.
Paschoal, L. N., de Oliveira, M. M., and Chicon, P. M. M. (2018). A chatterbot sensitive to student’s context to help on software engineering education. In Latin American Computer Conference, pages 839–848.
Paschoal, L. N., Melo, S. M., Neves, V. d. O., Conte, T. U., and de SOUZA, S. d. R. S. (2023). An experimental study on a conversational agent in software testing lessons. Informatics in Education, 22(1):99–120.
Paschoal, L. N., Turci, L. F., Conte, T. U., and Souza, S. R. (2019). Towards a conversational agent to support the software testing education. In Brazilian Symposium on Software Engineering, pages 57–66.
Pérez-Soler, S., Guerra, E., and De Lara, J. (2020). Model-driven chatbot development. In International Conference on Conceptual Modeling, pages 207–222. Springer.
Pérez-Soler, S., Juárez-Puerta, S., Guerra, E., and de Lara, J. (2021). Choosing a chatbot development tool. IEEE Software, 38(4):94–103.
Roca, M. D. L., Chan, M. M., Garcia-Cabot, A., Garcia-Lopez, E., and Amado-Salvatierra, H. (2024). The impact of a chatbot working as an assistant in a course for supporting student learning and engagement. Computer Applications in Engineering Education, 32(5):e22750.
Wohlin, C., Runeson, P., Hst, M., Ohlsson, M. C., Regnell, B., and Wessln, A. (2012). Experimentation in Software Engineering. Springer Publishing Company, Incorporated.
Publicado
24/11/2025
Como Citar
MUSICO, Thiago; MELO, Silvana Morita; PASCHOAL, Leo Natan; SOUZA, Simone do Rocio Senger de.
Um chatbot treinado para entender intenções de estudantes em diálogos sobre teste de software. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 36. , 2025, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 873-886.
DOI: https://doi.org/10.5753/sbie.2025.12704.
