Automatic Generation of Bug Reports Using Large Language Models: An Evaluation in a Software Institute - Chaves, Lennon; Gonzaga, Davi; Tiago, Leonardo; Silva, Ana Paula; Oliveira, Flávia (Anais do Simpósio Brasileiro de Qualidade de Software (SBQS); 2025: Anais do XXIV Simpósio Brasileiro de Qualidade de Software, 2025)
Context: During software development, the test team is responsible for verifying if the requirements were correctly implemented. In case the software does not behave as expected, the tester must report the bug to the development team. For the problem to be assertively and quickly, it is important that the bug report is complete, containing all the information needed to fix the bug. This study was conducted in the context of a test team that acts in a software institute, and is responsible for testing mobile devices. Problem: Among their activities, the test team must write bug reports. However, given a high amount of test requests, consequently there will be many bug reports, which will take more time and effort. Goal: With the goal of automating the bug report process, we developed a system that automatically generates bug reports based on the Text-to-Text Transfer Transformer (T5) Large Language Model (LLM). To generate a bug report, the tester must input a prompt containing a brief description of the bug. Method: We also performed an experiment with 8 members from the test team that write bug reports daily, so as to measure their perceptions regarding the solution we developed. The participants used the system and evaluated its outputs for the generation of 5 distinct bug reports. Results: Results showed a high acceptance rate for using the system within the test team, with 3 of the 5 bug types included in the experiment having 87.5% of outputs considered valid or partially valid. Furthermore, participants highlighted the ease of use, the efficacy and quality of writing in the bug reports generated by the system. However, they also noted that the system still needs to be adjusted to work for more types of issues, thus reducing the need for manual fixes. Conclusions: We conclude, then, that it is possible to use LLMs to automatically generate bug reports, however it is still indispensable to have a human review the report after it is generated.