Evaluating the Effectiveness and Cost-Efficiency of Large Language Models in Automated Unit Test Generation - Lira, Werney Ayala Luz; SantosNeto, Pedro de Alcântara dos; Avelino, Guilherme Amaral; Osório, Luiz Fernando Mendes (Anais do Simpósio Brasileiro de Qualidade de Software (SBQS); 2025: Anais do XXIV Simpósio Brasileiro de Qualidade de Software, 2025)
Software quality plays a crucial role in the development process, and one of the main strategies to ensure it is through software testing. However, testing can be a costly and time-consuming task, leading researchers to explore ways to automate it. This study analyzes the ability of large language models (LLMs) to generate unit tests automatically, evaluating their performance based on four key criteria: line coverage, branch coverage, hit rate, and cost. To achieve this, unit tests were generated using three different input formats: (i) source code only, (ii) source code with a docstring, and (iii) a detailed prompt with step-by-step instructions. The results show that all models produce tests with high line and branch coverage. However, hit rates vary depending on the input format. The lowest hit rate was 72.61%, obtained by ChatGPT-4o Mini when provided with only the source code. The highest hit rate was 90.69%, achieved by Gemini 1.5 Pro when given source code with a docstring, a performance close to the 96.81% hit rate of the dataset’s reference tests. The findings indicate that more affordable models, such as Gemini 1.5 Flash and GPT-4o Mini, can achieve competitive results when optimized prompts are used, making them cost-effective alternatives for unit test generation.