Rubrica Analítica para Avaliação de Artefatos de Teste de Desempenho Gerados por LLMs: Um Estudo Exploratório
Resumo
Modelos de Linguagem de Grande Escala (Large Language Models, LLMs) têm sido investigados como apoio à geração de artefatos de teste de software, sobretudo em tarefas funcionais, como cenários BDD, casos de teste a partir de requisitos e planos de QA. Entretanto, cenários e casos de teste de desempenho exigem propriedades adicionais: escolha adequada do tipo de ensaio, definição de carga, preservação de fluxos e perfis de usuário, métricas observáveis, critérios numéricos de aceitação e viabilidade de execução. Este artigo apresenta uma rubrica analítica para avaliar a validade e a qualidade de artefatos de teste de desempenho gerados por LLMs. A rubrica contempla oito dimensões: correção técnica, rastreabilidade ao requisito, adequação do tipo de teste, coerência da carga, critérios numéricos de aceitação, completude, viabilidade de execução e clareza. O estudo é estruturado como uma avaliação empírica exploratória com duas condições de prompting: uma condição Baseline, na qual o modelo recebe o requisito, o contexto técnico e o template de saída; e uma condição Rubric-aware, na qual recebe os mesmos insumos acrescidos dos critérios operacionais da rubrica. Foram gerados 36 artefatos a partir de seis requisitos, três configurações de modelo e duas condições de prompting. Após adjudicação, todos os artefatos atenderam aos critérios invalidantes no contexto técnico fornecido aos modelos, mas a viabilidade de execução concentrou as notas mais baixas. Os resultados indicam que a rubrica é útil para revelar diferenças entre validade mínima, completude textual e operacionalização executável.
Palavras-chave:
Testes de Desempenho, Modelos de Linguagem de Grande Escala, Rubrica Analítica, Requisitos Não Funcionais, Geração de Casos de Teste
Referências
Waleed Abdeen, Xingru Chen, and Michael Unterkalmsteiner. 2023. An approach for performance requirements verification and test environments generation. Requirements Engineering 28, 1 (2023), 117–144.
Sabinakhon Akbarova, Felix Dobslaw, and Robert Feldt. 2026. Understanding on the Edge: LLM-generated Boundary Test Explanations. arXiv:2601.22791 [cs.SE] [link]
Chetan Arora, Tomas Herda, and Verena Homm. 2024. Generating Test Scenarios from NL Requirements using Retrieval-Augmented LLMs: An Industrial Study. In Proceedings of the 2024 IEEE/ACM 32nd International Requirements Engineering Conference (RE). IEEE, 240–251.
Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Junda He, Christoph Treude, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Cristina Martinez Montes, Lutz Prechelt, Paul Ralph, Rijnard van Tonder, and Stefan Wagner. 2025. Guidelines for Empirical Studies in Software Engineering involving Large Language Models. arXiv:2508.15503 [cs.SE] DOI: 10.48550/arXiv.2508.15503
Nathalia Barbosa, Paulo Borba, and Leuson Da Silva. 2025. Detecção de Conflitos Semânticos com Testes Gerados por LLM. In Anais do X SAST (SAST 2025). Sociedade Brasileira de Computaç ao, Porto Alegre, RS, Brasil, 18–27.
Ermanno Battista, Sergio Di Martino, Sergio Di Meglio, Fabio Scippacercola, and Luigi Libero Lucio Starace. 2023. E2E-Loader: A Framework to Support Performance Testing of Web Applications. In Proceedings of the 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 351–361. DOI: 10.1109/ICST57152.2023.00040
Shreya Bhatia, Tarushi Gandhi, Dhruv Kumar, and Pankaj Jalote. 2024. System Test Case Design from Requirements Specifications: Insights and Challenges of Using ChatGPT. arXiv:2412.03693 [cs.SE] [link]
Jessica Díaz, Jorge Pérez, Carolina Gallardo, and Ángel González-Prieto. 2023. Applying Inter-Rater Reliability and Agreement in collaborative Grounded Theory studies in software engineering. Journal of Systems and Software 195 (2023), 111520.
Nayse Fagundes and Leopoldo Teixeira. 2025. Evaluating LLMs for Multimodal GUI Test Generation in Android Applications. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (SAST 2025). Sociedade Brasileira de Computaç ao, Porto Alegre, RS, Brasil, 28–36.
Daniel Graziotin, Per Lenberg, Robert Feldt, and StefanWagner. 2021. Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines. ACM Transactions on Software Engineering and Methodology 31, 1 (2021), 1–36.
Navid Bin Hasan, Md. Ashraful Islam, Junaed Younus Khan, Sanjida Senjik, and Anindya Iqbal. 2025. Automatic High-Level Test Case Generation using Large Language Models. In Proceedings of the 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE Computer Society, Los Alamitos, CA, USA, 674–685.
Benedikt Hauptmann, Maximilian Junker, Sebastian Eder, Lars Heinemann, Rudolf Vaas, and Peter Braun. 2013. Hunting for Smells in Natural Language Tests. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 1217–1220.
International Software Testing Qualifications Board. 2018. Certified Tester Foundation Level Specialist: Performance Testing Syllabus. [link] Acessado em: 20 maio 2026.
Feng Lin, Dong Jae Kim, Zhenhao Li, Jinqiu Yang, and Tse-Hsun Chen. 2025. RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation. arXiv:2503.22851 [cs.SE] [link]
Satoshi Masuda, Satoshi Kouzawa, Kyousuke Sezai, Hidetoshi Suhara, Yasuaki Hiruta, and Kunihiro Kudou. 2025. Generating High-Level Test Cases from Requirements using LLM: An Industry Study. arXiv:2510.03641 [cs.SE] [link]
Evin Aslan Oğuz and Jochen Malte Kuester. 2024. A Comparative Analysis of ChatGPT-Generated and Human-Written Use Case Descriptions. In Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems (MODELS 2024) (MODELS Companion ’24). ACM, New York, NY, USA, 533–540.
Ruwei Pan, Yakun Zhang, Qingyuan Liang, Yueheng Zhu, Chao Liu, Lu Zhang, and Hongyu Zhang. 2026. Toward Functional and Non-Functional Evaluation of Application-Level Code Generation. arXiv:2602.03462 [cs.SE] [link]
John Pangas, Suhaib Mujahid, Ahmad Abdellatif, and Marco Castelluccio. 2026. Using LLMs to Bridge the Gaps in QA Test Plans at Firefox. IEEE Software 43, 1 (2026), 41–47.
Delip Rao and Chris Callison-Burch. 2026. Autorubric: Unifying Rubric-based LLM Evaluation. arXiv:2603.00077 [cs.CL] [link]
Amila Rathnayake, Mojtaba Shahin, and Golnoush Abaei. 2026. Behaviour Driven Development Scenario Generation with Large Language Models. arXiv:2603.04729 [cs.SE] [link]
Henning Schulz, Dušan Okanović, André van Hoorn, Vincenzo Ferme, and Cesare Pautasso. 2019. Behavior-driven Load Testing Using Contextual Knowledge—Approach and Experiences. In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering. ACM, 265–272.
Henning Schulz, Dušan Okanović, André van Hoorn, and Petr Tůma. 2021. Context-tailored Workload Model Generation for Continuous Representative Load Testing. In Proceedings of the 2021 ACM/SPEC International Conference on Performance Engineering. ACM, 21–32.
Wayne D. Smith. 2000. TPC–W: Benchmarking an Ecommerce Solution. Technical Report Revision 1.2. Transaction Processing Performance Council. [link]
Elvys Soares, Manoel Aranda, Naelson Oliveira, Márcio Ribeiro, Rohit Gheyi, Emerson Souza, Ivan Machado, André Santos, Baldoino Fonseca, and Rodrigo Bonifácio. 2023. Manual Tests Do Smell! Cataloging and Identifying Natural Language Test Smells. In 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1–11.
Xin Sun, Daniel Ståhl, Kristian Sandahl, and Christoph Kessler. 2026. Quality Assurance of LLM-generated Code: Addressing Non-functional Quality Characteristics. Journal of Systems and Software 238 (2026), 112885.
IrdinaWanda Syahputri, Eko K. Budiardjo, and Panca O. Hadi Putra. 2025. Unlocking the Potential of the Prompt Engineering Paradigm in Software Engineering: A Systematic Literature Review. AI (Basel) 6, 9 (2025), 206.
Huynh Khanh Vi Tran, Nauman bin Ali, Michael Unterkalmsteiner, Jürgen Börstler, and Panagiota Chatzipetrou. 2025. Quality Attributes of Test Cases and Test Suites – Importance & Challenges from Practitioners’ Perspectives. Software Quality Journal 33 (2025), 123–160.
Muhammad Usman, Nauman Bin Ali, and Claes Wohlin. 2023. A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering. e-Informatica Software Engineering Journal 17, 1 (2023), 230105.
Christian Vögele, André van Hoorn, Eike Schulz, Wilhelm Hasselbring, and Helmut Krcmar. 2018. WESSBAS: extraction of probabilistic workload specifications for load testing and performance prediction—a model-driven approach for session-based application systems. Software and Systems Modeling 17, 2 (2018), 443–477.
ShihaiWang and Tao Chen. 2025. Light over Heavy: Automated Performance Requirements Quantification with Linguistic Inducement. arXiv:2511.03421 [cs.SE] [link]
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 [cs.SE] [link]
Zhiyi Xue, Liangguo Li, Senyue Tian, Xiaohong Chen, Pingping Li, Liangyu Chen, Tingting Jiang, and Min Zhang. 2024. LLM4Fin: Fully Automating LLM-Powered Test Case Generation for FinTech Software Acceptance Testing. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2024) (ISSTA ’24). ACM, New York, NY, USA, 1643–1655.
Sabinakhon Akbarova, Felix Dobslaw, and Robert Feldt. 2026. Understanding on the Edge: LLM-generated Boundary Test Explanations. arXiv:2601.22791 [cs.SE] [link]
Chetan Arora, Tomas Herda, and Verena Homm. 2024. Generating Test Scenarios from NL Requirements using Retrieval-Augmented LLMs: An Industrial Study. In Proceedings of the 2024 IEEE/ACM 32nd International Requirements Engineering Conference (RE). IEEE, 240–251.
Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chunyang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Junda He, Christoph Treude, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Cristina Martinez Montes, Lutz Prechelt, Paul Ralph, Rijnard van Tonder, and Stefan Wagner. 2025. Guidelines for Empirical Studies in Software Engineering involving Large Language Models. arXiv:2508.15503 [cs.SE] DOI: 10.48550/arXiv.2508.15503
Nathalia Barbosa, Paulo Borba, and Leuson Da Silva. 2025. Detecção de Conflitos Semânticos com Testes Gerados por LLM. In Anais do X SAST (SAST 2025). Sociedade Brasileira de Computaç ao, Porto Alegre, RS, Brasil, 18–27.
Ermanno Battista, Sergio Di Martino, Sergio Di Meglio, Fabio Scippacercola, and Luigi Libero Lucio Starace. 2023. E2E-Loader: A Framework to Support Performance Testing of Web Applications. In Proceedings of the 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 351–361. DOI: 10.1109/ICST57152.2023.00040
Shreya Bhatia, Tarushi Gandhi, Dhruv Kumar, and Pankaj Jalote. 2024. System Test Case Design from Requirements Specifications: Insights and Challenges of Using ChatGPT. arXiv:2412.03693 [cs.SE] [link]
Jessica Díaz, Jorge Pérez, Carolina Gallardo, and Ángel González-Prieto. 2023. Applying Inter-Rater Reliability and Agreement in collaborative Grounded Theory studies in software engineering. Journal of Systems and Software 195 (2023), 111520.
Nayse Fagundes and Leopoldo Teixeira. 2025. Evaluating LLMs for Multimodal GUI Test Generation in Android Applications. In Anais do X Simpósio Brasileiro de Testes de Software Sistemático e Automatizado (SAST 2025). Sociedade Brasileira de Computaç ao, Porto Alegre, RS, Brasil, 28–36.
Daniel Graziotin, Per Lenberg, Robert Feldt, and StefanWagner. 2021. Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines. ACM Transactions on Software Engineering and Methodology 31, 1 (2021), 1–36.
Navid Bin Hasan, Md. Ashraful Islam, Junaed Younus Khan, Sanjida Senjik, and Anindya Iqbal. 2025. Automatic High-Level Test Case Generation using Large Language Models. In Proceedings of the 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE Computer Society, Los Alamitos, CA, USA, 674–685.
Benedikt Hauptmann, Maximilian Junker, Sebastian Eder, Lars Heinemann, Rudolf Vaas, and Peter Braun. 2013. Hunting for Smells in Natural Language Tests. In 2013 35th International Conference on Software Engineering (ICSE). IEEE, 1217–1220.
International Software Testing Qualifications Board. 2018. Certified Tester Foundation Level Specialist: Performance Testing Syllabus. [link] Acessado em: 20 maio 2026.
Feng Lin, Dong Jae Kim, Zhenhao Li, Jinqiu Yang, and Tse-Hsun Chen. 2025. RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation. arXiv:2503.22851 [cs.SE] [link]
Satoshi Masuda, Satoshi Kouzawa, Kyousuke Sezai, Hidetoshi Suhara, Yasuaki Hiruta, and Kunihiro Kudou. 2025. Generating High-Level Test Cases from Requirements using LLM: An Industry Study. arXiv:2510.03641 [cs.SE] [link]
Evin Aslan Oğuz and Jochen Malte Kuester. 2024. A Comparative Analysis of ChatGPT-Generated and Human-Written Use Case Descriptions. In Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems (MODELS 2024) (MODELS Companion ’24). ACM, New York, NY, USA, 533–540.
Ruwei Pan, Yakun Zhang, Qingyuan Liang, Yueheng Zhu, Chao Liu, Lu Zhang, and Hongyu Zhang. 2026. Toward Functional and Non-Functional Evaluation of Application-Level Code Generation. arXiv:2602.03462 [cs.SE] [link]
John Pangas, Suhaib Mujahid, Ahmad Abdellatif, and Marco Castelluccio. 2026. Using LLMs to Bridge the Gaps in QA Test Plans at Firefox. IEEE Software 43, 1 (2026), 41–47.
Delip Rao and Chris Callison-Burch. 2026. Autorubric: Unifying Rubric-based LLM Evaluation. arXiv:2603.00077 [cs.CL] [link]
Amila Rathnayake, Mojtaba Shahin, and Golnoush Abaei. 2026. Behaviour Driven Development Scenario Generation with Large Language Models. arXiv:2603.04729 [cs.SE] [link]
Henning Schulz, Dušan Okanović, André van Hoorn, Vincenzo Ferme, and Cesare Pautasso. 2019. Behavior-driven Load Testing Using Contextual Knowledge—Approach and Experiences. In Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering. ACM, 265–272.
Henning Schulz, Dušan Okanović, André van Hoorn, and Petr Tůma. 2021. Context-tailored Workload Model Generation for Continuous Representative Load Testing. In Proceedings of the 2021 ACM/SPEC International Conference on Performance Engineering. ACM, 21–32.
Wayne D. Smith. 2000. TPC–W: Benchmarking an Ecommerce Solution. Technical Report Revision 1.2. Transaction Processing Performance Council. [link]
Elvys Soares, Manoel Aranda, Naelson Oliveira, Márcio Ribeiro, Rohit Gheyi, Emerson Souza, Ivan Machado, André Santos, Baldoino Fonseca, and Rodrigo Bonifácio. 2023. Manual Tests Do Smell! Cataloging and Identifying Natural Language Test Smells. In 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1–11.
Xin Sun, Daniel Ståhl, Kristian Sandahl, and Christoph Kessler. 2026. Quality Assurance of LLM-generated Code: Addressing Non-functional Quality Characteristics. Journal of Systems and Software 238 (2026), 112885.
IrdinaWanda Syahputri, Eko K. Budiardjo, and Panca O. Hadi Putra. 2025. Unlocking the Potential of the Prompt Engineering Paradigm in Software Engineering: A Systematic Literature Review. AI (Basel) 6, 9 (2025), 206.
Huynh Khanh Vi Tran, Nauman bin Ali, Michael Unterkalmsteiner, Jürgen Börstler, and Panagiota Chatzipetrou. 2025. Quality Attributes of Test Cases and Test Suites – Importance & Challenges from Practitioners’ Perspectives. Software Quality Journal 33 (2025), 123–160.
Muhammad Usman, Nauman Bin Ali, and Claes Wohlin. 2023. A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering. e-Informatica Software Engineering Journal 17, 1 (2023), 230105.
Christian Vögele, André van Hoorn, Eike Schulz, Wilhelm Hasselbring, and Helmut Krcmar. 2018. WESSBAS: extraction of probabilistic workload specifications for load testing and performance prediction—a model-driven approach for session-based application systems. Software and Systems Modeling 17, 2 (2018), 443–477.
ShihaiWang and Tao Chen. 2025. Light over Heavy: Automated Performance Requirements Quantification with Linguistic Inducement. arXiv:2511.03421 [cs.SE] [link]
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 [cs.SE] [link]
Zhiyi Xue, Liangguo Li, Senyue Tian, Xiaohong Chen, Pingping Li, Liangyu Chen, Tingting Jiang, and Min Zhang. 2024. LLM4Fin: Fully Automating LLM-Powered Test Case Generation for FinTech Software Acceptance Testing. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2024) (ISSTA ’24). ACM, New York, NY, USA, 1643–1655.
Publicado
08/09/2026
Como Citar
GRILLO, Lucie; BITENCOURT, Graziela Espindola; BERNARDINO, Maicon; RODRIGUES, Elder Macedo.
Rubrica Analítica para Avaliação de Artefatos de Teste de Desempenho Gerados por LLMs: Um Estudo Exploratório. In: SIMPÓSIO BRASILEIRO DE TESTES DE SOFTWARE SISTEMÁTICO E AUTOMATIZADO (SAST), 11. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 99-108.
DOI: https://doi.org/10.5753/sast.2026.31187.
