Cronos: Automating Bug Report Validation with LLMs using RAG Approach
Resumo
In a testing team within a software institute, the validation of bug reports as valid or invalid is a critical but time-consuming activity. Performed manually, this validation is subject to errors, consumes effort and time, and competes with other activities of the testing team, especially when this team is under high demand, resulting in a backlog of bug reports to be validated. This study presents Cronos, an LLM-based tool with Retrieval Augmented Generation (RAG) that classifies bug reports as Valid or Not Valid, combining the retrieval of similar bug reports and relevant test cases. For this, we used 100 real bug reports collected and balanced between the Valid and Not Valid classes, submitted to the open source LLM models GPT, Qwen, and Gemma. Additionally, we used a also balanced knowledge base of 1526 bug reports and 133 test cases, retrieved through RAG with BERT embeddings and cosine similarity. We evaluated the models using the metrics Accuracy, Precision, Recall, and F1-Score, and statistical tests Cochran’s Q test and McNemar post-hoc tests with Holm adjustment. The results indicate that the Qwen model achieved superior performance compared to the others, with Accuracy of 82%, F1-Score of 79.55% for Valid and 83.93% for Not Valid. GPT presented a Recall of only 46% for Valid, classifying 54% of the valid bug reports as invalid, while Gemma achieved Precision of 100% for Valid but Recall of only 58%. Cochran’s Q test indicated a significant difference between the models with p-value = 0.0206, however the pairwise tests did not identify significant differences after the Holm adjustment. Therefore, the results of this study demonstrate that it is possible to use LLMs with RAG to automate the validation of bug reports in the industry, with Qwen being the most suitable model for balancing Precision and Recall for both classes.
Palavras-chave:
Bug Report Validation, Large Language Models, Retrieval Augmented Generation, Software Testing, Automated Classification
Referências
Victor R Basili. 1994. Goal, question, metric paradigm. Encyclopedia of software engineering 1 (1994), 528–532.
Oscar Chaparro. 2017. Improving Bug Reporting, Duplicate Detection, and Localization. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 421–424. DOI: 10.1109/ICSE-C.2017.27
Lennon Chaves, Davi Gonzaga, Leonardo Tiago, Ana Silva, and Flávia Oliveira. 2025. Automatic Generation of Bug Reports Using Large Language Models: An Evaluation in a Software Institute. In Anais do XXIV Simpósio Brasileiro de Qualidade de Software (São José dos Campos/SP). SBC, Porto Alegre, RS, Brasil, 337–345. DOI: 10.5753/sbqs.2025.13599
Lennon Chaves, Flávia Oliveira, and Leonardo Tiago. 2024. Automating issue reporting in software testing: Lessons learned from using the template generator tool. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 278–282.
Lennon Chaves, Flavia Oliveira, and Leonardo Tiago. 2024. Enhancing Automated Tools: Reporting Bugs with Bug Builder. In Anais do IX Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Curitiba/PR). SBC, Porto Alegre, RS, Brasil, 80–82. DOI: 10.5753/sast.2024.3674
Seojin Choi and Geunseok Yang. 2025. AgentReport: A Multi-Agent LLM Approach for Automated and Reproducible Bug Report Generation. Applied Sciences 15, 22 (2025). DOI: 10.3390/app152211931
William G Cochran. 1950. The comparison of percentages in matched samples. Biometrika 37, 3/4 (1950), 256–266.
Steven Davies and Marc Roper. 2014. What’s in a bug report?. In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1–10.
Janez Demšar. 2006. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research 7, Jan (2006), 1–30.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186.
Edna Dias Canedo and Bruno Cordeiro Mendes. 2020. Software requirements classification using machine learning algorithms. Entropy 22, 9 (2020), 1057.
Abdullah Dogan, Alev Mutlu, and Pinar Karagoz. 2016. Cosine Similarity-Based Pruning for Concept Discovery. In International Symposium on Computer and Information Sciences. Springer, 90–96.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining. 6491–6501.
Yuanrui Fan, Xin Xia, David Lo, and Ahmed E. Hassan. 2020. Chaff from the Wheat: Characterizing and Determining Valid Bug Reports. IEEE Transactions on Software Engineering 46, 5 (2020), 495–525. DOI: 10.1109/TSE.2018.2864217
Davi Gonzaga, Leonardo Tiago, Ana Silva, Flávia Oliveira, and Lennon Chaves. 2025. Improving Bug Reporting by Fine-Tuning the T5 Model: An Evaluation in a Software Industry. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Recife/PE). SBC, Porto Alegre, RS, Brasil, 141–143. DOI: 10.5753/sast.2025.13591
Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics (1979), 65–70.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. [n. d.]. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations.
Pieter Hooimeijer and Westley Weimer. 2007. Modeling bug report quality. In Proceedings of the 22nd IEEE/ACM international conference on Automated software engineering. 34–43.
Peter A Lachenbruch. 2005. McNemar Test. Encyclopedia of Biostatistics 5 (2005).
Alay Nascimento, Flávia Oliveira, Leonardo Tiago, and Lennon Chaves. 2025. Who Should Test the Requirement? A Comparative Study on Requirements Classification for Assigning Test Teams using the Pre-Trained Models. In Simpósio Brasileiro de Engenharia de Software (SBES). SBC, 671–677.
Mario Patrício, Silas Eufrásio, Anderson Uchôa, Lincoln S Rocha, Daniel Coutinho, Juliana Alves Pereira, Matheus Paixão, and Alessandro Garcia. 2025. 404: Civility Not Found? Evaluating the Effectiveness of Small Language Models in Detecting Incivility in GitHub Conversations. In Simpósio Brasileiro de Engenharia de Software (SBES). SBC, 304–314.
Rylan Schaeffer, Joshua Kazdan, and Yegor Denisov-Blanch. 2025. Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models. arXiv preprint arXiv:2506.13681 (2025).
Marina Sokolova and Guy Lapalme. 2009. A systematic analysis of performance measures for classification tasks. Information processing & management 45, 4 (2009), 427–437.
Jason Wei and Kai Zou. [n. d.]. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. ( [n. d.]).
Marcelo Serrano Zanetti, Ingo Scholtes, Claudio Juan Tessone, and Frank Schweitzer. 2013. Categorizing bugs with social networks: A case study on four open source software communities. In 2013 35th International Conference on Software Engineering (ICSE). 1032–1041. DOI: 10.1109/ICSE.2013.6606653
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. [n. d.]. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method. In The Twelfth International Conference on Learning Representations.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2016. Character-level Convolutional Networks for Text Classification. arXiv:1509.01626 [cs.LG] [link]
Zijie Zhao, Chenyuan Yang, Weidong Wang, Yihan Yang, Ziqi Zhang, and Lingming Zhang. 2026. AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection. arXiv preprint arXiv:2604.11950 (2026).
Thomas Zimmermann, Rahul Premraj, Nicolas Bettenburg, Sascha Just, Adrian Schroter, and Cathrin Weiss. 2010. What Makes a Good Bug Report? IEEE Transactions on Software Engineering 36, 5 (2010), 618–643. DOI: 10.1109/TSE.2010.63
Oscar Chaparro. 2017. Improving Bug Reporting, Duplicate Detection, and Localization. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). 421–424. DOI: 10.1109/ICSE-C.2017.27
Lennon Chaves, Davi Gonzaga, Leonardo Tiago, Ana Silva, and Flávia Oliveira. 2025. Automatic Generation of Bug Reports Using Large Language Models: An Evaluation in a Software Institute. In Anais do XXIV Simpósio Brasileiro de Qualidade de Software (São José dos Campos/SP). SBC, Porto Alegre, RS, Brasil, 337–345. DOI: 10.5753/sbqs.2025.13599
Lennon Chaves, Flávia Oliveira, and Leonardo Tiago. 2024. Automating issue reporting in software testing: Lessons learned from using the template generator tool. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 278–282.
Lennon Chaves, Flavia Oliveira, and Leonardo Tiago. 2024. Enhancing Automated Tools: Reporting Bugs with Bug Builder. In Anais do IX Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Curitiba/PR). SBC, Porto Alegre, RS, Brasil, 80–82. DOI: 10.5753/sast.2024.3674
Seojin Choi and Geunseok Yang. 2025. AgentReport: A Multi-Agent LLM Approach for Automated and Reproducible Bug Report Generation. Applied Sciences 15, 22 (2025). DOI: 10.3390/app152211931
William G Cochran. 1950. The comparison of percentages in matched samples. Biometrika 37, 3/4 (1950), 256–266.
Steven Davies and Marc Roper. 2014. What’s in a bug report?. In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 1–10.
Janez Demšar. 2006. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research 7, Jan (2006), 1–30.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186.
Edna Dias Canedo and Bruno Cordeiro Mendes. 2020. Software requirements classification using machine learning algorithms. Entropy 22, 9 (2020), 1057.
Abdullah Dogan, Alev Mutlu, and Pinar Karagoz. 2016. Cosine Similarity-Based Pruning for Concept Discovery. In International Symposium on Computer and Information Sciences. Springer, 90–96.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining. 6491–6501.
Yuanrui Fan, Xin Xia, David Lo, and Ahmed E. Hassan. 2020. Chaff from the Wheat: Characterizing and Determining Valid Bug Reports. IEEE Transactions on Software Engineering 46, 5 (2020), 495–525. DOI: 10.1109/TSE.2018.2864217
Davi Gonzaga, Leonardo Tiago, Ana Silva, Flávia Oliveira, and Lennon Chaves. 2025. Improving Bug Reporting by Fine-Tuning the T5 Model: An Evaluation in a Software Industry. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Recife/PE). SBC, Porto Alegre, RS, Brasil, 141–143. DOI: 10.5753/sast.2025.13591
Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics (1979), 65–70.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. [n. d.]. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations.
Pieter Hooimeijer and Westley Weimer. 2007. Modeling bug report quality. In Proceedings of the 22nd IEEE/ACM international conference on Automated software engineering. 34–43.
Peter A Lachenbruch. 2005. McNemar Test. Encyclopedia of Biostatistics 5 (2005).
Alay Nascimento, Flávia Oliveira, Leonardo Tiago, and Lennon Chaves. 2025. Who Should Test the Requirement? A Comparative Study on Requirements Classification for Assigning Test Teams using the Pre-Trained Models. In Simpósio Brasileiro de Engenharia de Software (SBES). SBC, 671–677.
Mario Patrício, Silas Eufrásio, Anderson Uchôa, Lincoln S Rocha, Daniel Coutinho, Juliana Alves Pereira, Matheus Paixão, and Alessandro Garcia. 2025. 404: Civility Not Found? Evaluating the Effectiveness of Small Language Models in Detecting Incivility in GitHub Conversations. In Simpósio Brasileiro de Engenharia de Software (SBES). SBC, 304–314.
Rylan Schaeffer, Joshua Kazdan, and Yegor Denisov-Blanch. 2025. Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models. arXiv preprint arXiv:2506.13681 (2025).
Marina Sokolova and Guy Lapalme. 2009. A systematic analysis of performance measures for classification tasks. Information processing & management 45, 4 (2009), 427–437.
Jason Wei and Kai Zou. [n. d.]. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. ( [n. d.]).
Marcelo Serrano Zanetti, Ingo Scholtes, Claudio Juan Tessone, and Frank Schweitzer. 2013. Categorizing bugs with social networks: A case study on four open source software communities. In 2013 35th International Conference on Software Engineering (ICSE). 1032–1041. DOI: 10.1109/ICSE.2013.6606653
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. [n. d.]. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method. In The Twelfth International Conference on Learning Representations.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2016. Character-level Convolutional Networks for Text Classification. arXiv:1509.01626 [cs.LG] [link]
Zijie Zhao, Chenyuan Yang, Weidong Wang, Yihan Yang, Ziqi Zhang, and Lingming Zhang. 2026. AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection. arXiv preprint arXiv:2604.11950 (2026).
Thomas Zimmermann, Rahul Premraj, Nicolas Bettenburg, Sascha Just, Adrian Schroter, and Cathrin Weiss. 2010. What Makes a Good Bug Report? IEEE Transactions on Software Engineering 36, 5 (2010), 618–643. DOI: 10.1109/TSE.2010.63
Publicado
08/09/2026
Como Citar
NASCIMENTO, Alay; SILVA, Ana Paula; CHAVES, Lennon.
Cronos: Automating Bug Report Validation with LLMs using RAG Approach. In: SIMPÓSIO BRASILEIRO DE TESTES DE SOFTWARE SISTEMÁTICO E AUTOMATIZADO (SAST), 11. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 1-8.
DOI: https://doi.org/10.5753/sast.2026.30180.
