Evaluating LLM-Generated Unit Tests with Mutation Testing: ChatGPT vs DeepSeek - Cabral, Pedro Fernando Marinho; Arruda, João Pedro Souza; Souza, Cleidson Ronald Botelho de; Pinto, Victor Hugo Santiago Costa (Anais do Simpósio Brasileiro de Qualidade de Software (SBQS); 2025: Anais do XXIV Simpósio Brasileiro de Qualidade de SoftwareDO - 10.5753/sbqs.2025.13792, 2025)
Recent advances in Large Language Models (LLMs) have driven significant progress in automating software testing, particularly in generating unit tests. However, the effectiveness of these models in detecting real defects through mutation testing remains underexplored in the literature. This study aims to address this gap by evaluating the performance of ChatGPT (GPT-4o) and DeepSeek V3 in generating unit tests for six Java classes from the Defects4J dataset, covering different levels of cyclomatic complexity. The main objective is to investigate the ability of LLMs to maximize mutant coverage and elimination, while also analyzing the impact of code complexity and semantic factors related to execution failures. The methodology involved generating tests via structured prompts, executing them 5 times per class for both models, and performing quantitative analysis based on Mutation Coverage (MC) and Mutation Score (MS), as well as qualitative analysis of runtime failures. Results indicate that DeepSeek exhibits greater stability and effectiveness in eliminating mutants, whereas ChatGPT demonstrates broader applicability by producing valid test suites for a wider range of classes. Moreover, no significant correlation was found between cyclomatic complexity and compilation success, with failures primarily linked to semantic limitations of the models. This study presents both quantitative and qualitative evidence on the application of LLMs for automated test generation, offering insights for future AI-driven test engineering strategies.