Teste de Mutação em System Prompts para Avaliação de Conjuntos de Testes de Chatbots Baseados em LLMs
Resumo
Ao testar cenários conversacionais em chatbots baseados em Large Language Models (LLMs), avaliar a adequação de um conjunto de testes é uma tarefa desafiadora, especialmente quando o comportamento do sistema é definido por linguagem natural (no system prompt) e não estritamente por código. Este trabalho propõe a aplicação do teste de mutação a chatbots configurados integralmente por um system prompt: os operadores de mutação atuam diretamente sobre o prompt, alterando regras e restrições para gerar variantes defeituosas (mutantes) do chatbot. A adequação do conjunto de testes é quantificada por um escore de mutação, apurado com um oráculo duplo que combina verificação determinística (Botium) e semântica (LLM-as-a-judge). A abordagem é demonstrada em um estudo de caso com um chatbot de agendamento de exames médicos. Os resultados evidenciam que o escore de mutação bruto (0,86) superestima a adequação do conjunto de testes: ao inspecionar as justificativas do oráculo, apenas 46 das 103 mortes (44,7%) são coerentes com o defeito injetado, o que reduz o escore para 0,38. Conclui-se que, em sistemas não-determinísticos configurados por linguagem natural, o escore de mutação deve ser interpretado à luz de uma análise de coerência das mortes, e não isoladamente.
Palavras-chave:
Teste de Mutação, Operadores de Mutação, Chatbots
Referências
Paul Ammann and Jeff Offutt. 2016. Introduction to Software Testing (2nd ed.). Cambridge University Press, Cambridge.
Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. 2025. Non-Determinism of "Deterministic" LLM Settings. arXiv:2408.04667 [cs.CL] [link]
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin,Wenbo Su, Tiezheng Ge, Bo Zheng, andWanli Ouyang. 2024. MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand, 7421–7454. DOI: 10.18653/v1/2024.acl-long.401
Botium. 2026. Botium Documentation. [link]. Accessed: 2026-06-26.
Josip Božić. 2021. Ontology-based metamorphic testing for chatbots. Software Quality Journal 30 (2021), 227–251. DOI: 10.1007/s11219-020-09544-9
Sergio Bravo-Santos, Esther Guerra, and Juan Lara. 2020. Testing Chatbots with Charm. Springer, Cham, Cham, Switzerland, 426–438. DOI: 10.1007/978-3-030-58793-2_34
Jordi Cabot, Loli Burgueno, Robert Clariso, Gwendal Daniel, Jorge Perianez-Pascual, and Roberto Rodriguez-Echeverria. 2021. Testing challenges for NLPintensive bots . In 2021 IEEE/ACM Third InternationalWorkshop on Bots in Software Engineering (BotSE). IEEE Computer Society, Los Alamitos, CA, USA, 31–34. DOI: 10.1109/BotSE52550.2021.00014
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132 [cs.AI] [link]
Nadezhda Chirkova, Tunde Oluwaseyi Ajayi, Seth Aycock, Zain Muhammad Mujahid, Vladana Perlić, Ekaterina Borisova, and Markarit Vartampetian. 2025. LLM-as-a-qualitative-judge: automating error analysis in natural language generation. arXiv:2506.09147 [cs.CL] [link]
Diego Clerissi, Elena Masserini, Daniela Micucci, and Leonardo Mariani. 2026. Automated Testing of Task-based Chatbots: How Far Are We? arXiv:2602.13072 [cs.SE] [link]
R.A. DeMillo, R.J. Lipton, and F.G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 4 (1978), 34–41. DOI: 10.1109/CM.1978.218136
Google. 2026. Gemini 3.5 Flash. [link]. Google AI for Developers. Last updated: 2026-06-24. Accessed: 2026-06-28.
Google Cloud. 2026. Dialogflow Documentation. [link]. Accessed: 2026-06-26.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A Survey on LLM-as-a-Judge. arXiv:2411.15594 [cs.CL] [link]
Jonathan Guichard, Elayne Ruane, Ross Smith, Dan Bean, and Anthony Ventresque. 2019. Assessing the Robustness of Conversational Agents using Paraphrases . In 2019 IEEE International Conference On Artificial Intelligence Testing (AITest). IEEE Computer Society, Los Alamitos, CA, USA, 55–62. DOI: 10.1109/AITest.2019.000-7
Rajarshi Haldar and Julia Hockenmaier. 2025. Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks. In Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, Suzhou, China, 24986–25004. DOI: 10.18653/v1/2025.findings-emnlp.1361
Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. DOI: 10.1109/TSE.2010.62
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with More Persuasive LLMs Leads to More Truthful Answers. arXiv:2402.06782 [cs.AI] [link]
Mugeng Liu, Siqi Zhong, Weichen Bi, Yixuan Zhang, Zhiyang Chen, Zhenpeng Chen, Xuanzhe Liu, and Yun Ma. 2026. A First Look at Bugs in LLM Inference Engines. arXiv:2506.09713 [cs.SE] [link]
Glenford J. Myers, Corey Sandler, and Tom Badgett. 2012. The art of software testing (3rd ed ed.). John Wiley & Sons, Hoboken, N.J.
OpenAI. 2026. GPT-5.4 mini. [link]. OpenAI API Documentation. Accessed: 2026-06-28.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM Evaluators Recognize and Favor Their Own Generations. arXiv:2404.13076 [cs.CL] [link]
Yanzhao Qin, Tao Zhang, Tao Zhang, Yanjun Shen, Wenjing Luo, Haoze Sun, Yan Zhang, Yujing Qiao, Weipeng Chen, Zenan Zhou, Wentao Zhang, and Bin Cui. 2024. SysBench: Can Large Language Models Follow System Messages? arXiv:2408.10943 [cs.CL] [link]
Rasa Technologies GmbH. 2026. Rasa Documentation. [link]. Accessed: 2026-06-26.
Yuqi Tang, Kehua Feng, Yunfeng Wang, Zhiwen Chen, Chengfei Lv, Gang Yu, Qiang Zhang, Keyan Ding, and Huajun Chen. 2026. Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judges. arXiv:2508.00454 [cs.CL] [link]
Zhenwei Tang, Zhaoyan Liu, Rasa Hosseinzadeh, Tongzi Wu, Keyvan Golestan, and Jesse C. Cresswell. 2026. RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator. arXiv:2605.21748 [cs.CL] [link]
Michael Ferdinando Urrico, Diego Clerissi, and Leonardo Mariani. 2024. MutaBot: A Mutation Testing Approach for Chatbots. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (Lisbon, Portugal) (ICSE-Companion ’24). Association for Computing Machinery, New York, NY, USA, 79–83. DOI: 10.1145/3639478.3640032
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2024. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. arXiv:2410.02736 [cs.CL] [link]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL] [link]
Berk Atil, Sarp Aykent, Alexa Chittams, Lisheng Fu, Rebecca J. Passonneau, Evan Radcliffe, Guru Rajan Rajagopal, Adam Sloan, Tomasz Tudrej, Ferhan Ture, Zhe Wu, Lixinyu Xu, and Breck Baldwin. 2025. Non-Determinism of "Deterministic" LLM Settings. arXiv:2408.04667 [cs.CL] [link]
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin,Wenbo Su, Tiezheng Ge, Bo Zheng, andWanli Ouyang. 2024. MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Bangkok, Thailand, 7421–7454. DOI: 10.18653/v1/2024.acl-long.401
Botium. 2026. Botium Documentation. [link]. Accessed: 2026-06-26.
Josip Božić. 2021. Ontology-based metamorphic testing for chatbots. Software Quality Journal 30 (2021), 227–251. DOI: 10.1007/s11219-020-09544-9
Sergio Bravo-Santos, Esther Guerra, and Juan Lara. 2020. Testing Chatbots with Charm. Springer, Cham, Cham, Switzerland, 426–438. DOI: 10.1007/978-3-030-58793-2_34
Jordi Cabot, Loli Burgueno, Robert Clariso, Gwendal Daniel, Jorge Perianez-Pascual, and Roberto Rodriguez-Echeverria. 2021. Testing challenges for NLPintensive bots . In 2021 IEEE/ACM Third InternationalWorkshop on Bots in Software Engineering (BotSE). IEEE Computer Society, Los Alamitos, CA, USA, 31–34. DOI: 10.1109/BotSE52550.2021.00014
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132 [cs.AI] [link]
Nadezhda Chirkova, Tunde Oluwaseyi Ajayi, Seth Aycock, Zain Muhammad Mujahid, Vladana Perlić, Ekaterina Borisova, and Markarit Vartampetian. 2025. LLM-as-a-qualitative-judge: automating error analysis in natural language generation. arXiv:2506.09147 [cs.CL] [link]
Diego Clerissi, Elena Masserini, Daniela Micucci, and Leonardo Mariani. 2026. Automated Testing of Task-based Chatbots: How Far Are We? arXiv:2602.13072 [cs.SE] [link]
R.A. DeMillo, R.J. Lipton, and F.G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 4 (1978), 34–41. DOI: 10.1109/CM.1978.218136
Google. 2026. Gemini 3.5 Flash. [link]. Google AI for Developers. Last updated: 2026-06-24. Accessed: 2026-06-28.
Google Cloud. 2026. Dialogflow Documentation. [link]. Accessed: 2026-06-26.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A Survey on LLM-as-a-Judge. arXiv:2411.15594 [cs.CL] [link]
Jonathan Guichard, Elayne Ruane, Ross Smith, Dan Bean, and Anthony Ventresque. 2019. Assessing the Robustness of Conversational Agents using Paraphrases . In 2019 IEEE International Conference On Artificial Intelligence Testing (AITest). IEEE Computer Society, Los Alamitos, CA, USA, 55–62. DOI: 10.1109/AITest.2019.000-7
Rajarshi Haldar and Julia Hockenmaier. 2025. Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks. In Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics, Suzhou, China, 24986–25004. DOI: 10.18653/v1/2025.findings-emnlp.1361
Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. DOI: 10.1109/TSE.2010.62
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with More Persuasive LLMs Leads to More Truthful Answers. arXiv:2402.06782 [cs.AI] [link]
Mugeng Liu, Siqi Zhong, Weichen Bi, Yixuan Zhang, Zhiyang Chen, Zhenpeng Chen, Xuanzhe Liu, and Yun Ma. 2026. A First Look at Bugs in LLM Inference Engines. arXiv:2506.09713 [cs.SE] [link]
Glenford J. Myers, Corey Sandler, and Tom Badgett. 2012. The art of software testing (3rd ed ed.). John Wiley & Sons, Hoboken, N.J.
OpenAI. 2026. GPT-5.4 mini. [link]. OpenAI API Documentation. Accessed: 2026-06-28.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM Evaluators Recognize and Favor Their Own Generations. arXiv:2404.13076 [cs.CL] [link]
Yanzhao Qin, Tao Zhang, Tao Zhang, Yanjun Shen, Wenjing Luo, Haoze Sun, Yan Zhang, Yujing Qiao, Weipeng Chen, Zenan Zhou, Wentao Zhang, and Bin Cui. 2024. SysBench: Can Large Language Models Follow System Messages? arXiv:2408.10943 [cs.CL] [link]
Rasa Technologies GmbH. 2026. Rasa Documentation. [link]. Accessed: 2026-06-26.
Yuqi Tang, Kehua Feng, Yunfeng Wang, Zhiwen Chen, Chengfei Lv, Gang Yu, Qiang Zhang, Keyan Ding, and Huajun Chen. 2026. Learning an Efficient Multi-Turn Dialogue Evaluator from Multiple LLM Judges. arXiv:2508.00454 [cs.CL] [link]
Zhenwei Tang, Zhaoyan Liu, Rasa Hosseinzadeh, Tongzi Wu, Keyvan Golestan, and Jesse C. Cresswell. 2026. RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator. arXiv:2605.21748 [cs.CL] [link]
Michael Ferdinando Urrico, Diego Clerissi, and Leonardo Mariani. 2024. MutaBot: A Mutation Testing Approach for Chatbots. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (Lisbon, Portugal) (ICSE-Companion ’24). Association for Computing Machinery, New York, NY, USA, 79–83. DOI: 10.1145/3639478.3640032
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2024. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. arXiv:2410.02736 [cs.CL] [link]
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL] [link]
Publicado
08/09/2026
Como Citar
BRAGA, Carlos E. C.; PAIVA, Sofia L. C..
Teste de Mutação em System Prompts para Avaliação de Conjuntos de Testes de Chatbots Baseados em LLMs. In: SIMPÓSIO BRASILEIRO DE TESTES DE SOFTWARE SISTEMÁTICO E AUTOMATIZADO (SAST), 11. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 248-257.
DOI: https://doi.org/10.5753/sast.2026.32381.
