Cassiopeia: A Retrieval-Augmented Generation Tool for Automated Test-Case Creation

• Flávia Camila Oliveira Sidia
• Leonardo Tiago Sidia
• Lennon Correa Chaves Sidia

DOI: https://doi.org/10.5753/sast.2026.30673

Resumo

This study, conducted within a software testing team that performs exploratory and functional testing, evaluates the feasibility of using artificial intelligence to automatically update and create test cases, thereby reducing the workload of the test case management. We built Cassiopeia, a tool that parses testers’ task submissions, formulates prompts for the large language model gpt-oss-120b, and supplies the test-case dataset context via Retrieval Augmented Generation (RAG) using the FAISS library for similarity search. In an experiment, 45 real change requests were processed; 16 generated test cases were independently rated by three testers on a 1–5 quality scale. Cassiopeia achieved 95.35% recall for test-case IDs, 53.32% BLEU-4, and 56.89% ROUGE-1. Overall, 58% of ratings indicated partial or full quality, 31% expressed disagreement, and 11% were uncertain. Results suggest the tool is promising but still requires complete task descriptions and human review before deployment.

Palavras-chave: Test Case Generator, RAG, LLM, Software Test, Requirements

Referências

Lennon Chaves, Flavia Oliveira, Leonardo Tiago, and Renata Castro. 2024. Benefits and Challenges of a Physical Device Farm for Automated Software Testing: A Case Study of the STELA Tool Implementation. In Anais do IX Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Curitiba/PR). SBC, Porto Alegre, RS, Brasil, 89–91. DOI: 10.5753/sast.2024.3880

Flávia Oliveira, Alay Nascimento, Ana Silva, Leonardo Tiago, and Lennon Chaves. 2025. Analysis of Contributions at a Software Institute through the Introduction of a Pre-Trained Model for Requirements Classification. In Anais do XXIV Simpósio Brasileiro de Qualidade de Software (São José dos Campos/SP). SBC, Porto Alegre, RS, Brasil, 356–364. DOI: 10.5753/sbqs.2025.13598

Flávia Oliveira, Ana Silva, Leonardo Tiago, and Lennon Chaves. 2025. Ensuring Test Case Coverage through Multilabel Requirements Classification: A Feasibility Study with Pre-Trained Models. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Recife/PE). SBC, Porto Alegre, RS, Brasil, 1–9. DOI: 10.5753/sast.2025.13597

Flávia Oliveira, Leonardo Tiago, and Lennon Chaves. 2025. The Influence of Using LLMs on the Activities of a Software Testing Team: An Industry Case. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (Recife/PE). SBC, Porto Alegre, RS, Brasil, 144–146. DOI: 10.5753/sast.2025.13592