Detecção de Inconsistências Lógicas em Casos de Teste por Meio de Extração Semântica e Análise de Alcançabilidade
Resumo
A identificação de inconsistências lógicas em artefatos de teste é um desafio crítico na engenharia de software, resultando frequentemente em retrabalho e elevados custos de manutenção. Revisões manuais tradicionais são morosas e suscetíveis a falhas, especialmente em sistemas de larga escala. Este trabalho propõe o framework de auditoria automática DILCAT, fundamentado em uma abordagem de decisão conservadora neuro-simbólica que integra a flexibilidade semântica de Sentence-Transformers com o poder preditivo de classificadores de Machine Learning supervisionados e Filtragem por Classes Gramaticais. Alinhada à norma ISO/IEC/IEEE 29119-3:2021, a metodologia propõe reduzir decisões heurísticas rígidas com uma classificação probabilística baseada em limiar de corte preventivo, capaz de categorizar as relações funcionais locais entre os casos de teste nas classes GAP, SEQUENTIAL_CHAIN e STANDALONE. Os resultados experimentais foram validados em um dataset orgânico e legado de 80 casos de teste da indústria de telecomunicações, os quais totalizam 406 instâncias de transição de estado, em conjunto com bases adicionais para testes fora de domínio. Os dados demonstram que a combinação de extração vetorial e aprendizado estatístico minimiza a incerteza linguística e mitiga falsos positivos funcionais. O ground truth foi estabelecido por três anotadores e validado pelo coeficiente estatístico Fleiss’ Kappa, alcançando um índice de 0,74, o que indica consistência entre os anotadores dos dados usados para validação e corrobora com a robustez do framework no tratamento de ruídos terminológicos e na facilitação da prevenção de intermitências antes da etapa de automação. Por fim, a avaliação experimental indicou que o sistema atinge uma acurácia global de 98% no conjunto avaliado, mantendo uma estabilidade macro de 94% de acurácia global durante os ensaios de validação cruzada.
Referências
Paul Ammann and Jeff Offutt. 2017. Introduction to Software Testing (2nd ed.). Cambridge University Press, Cambridge. DOI: 10.1017/9781316771273
Halima Ayenew and Mekonnen Wagaw. 2024. Software Test Case Generation Using Natural Language Processing (NLP): A Systematic Literature Review. Artificial Intelligence Evolution 5, 1 (Jan. 2024), 1–10. DOI: 10.37256/aie.5120243220
Christel Baier and Joost-Pieter Katoen. 2008. Principles of Model Checking. MIT Press, Cambridge, MA, USA.
Andreas Bauer, Martin Leucker, and Christian Schallhart. 2011. Runtime Verification for LTL and TLTL. ACM Transactions on Software Engineering and Methodology 20, 4, Article 14 (Sept. 2011), 64 pages. DOI: 10.1145/2000799.2000800
Adam Berger, Stephen A Della Pietra, and Vincent J Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational linguistics 22, 1 (1996), 39–71.
Matteo Biagiola, Andrea Stocco, Filippo Ricca, and Paolo Tonella. 2020. Dependency-Aware Web Test Generation. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) (2020-10). IEEE, IEEE, Los Alamitos, CA, USA, 175–185. DOI: 10.1109/ICST46399.2020.00027
Muhammad Shoaib Farooq, Uzma Omer, Amna Ramzan, Mansoor Ahmad Rasheed, and Zabihullah Atal. 2023. Behavior driven development: A systematic literature review. IEEE access 11 (2023), 88008–88024.
Jannik Fischbach, Julian Frattini, Andreas Vogelsang, Daniel Mendez, Michael Unterkalmsteiner, Andreas Wehrle, Pablo Restrepo Henao, Parisa Yousefi, Tedi Juricic, Jeannette Radduenz, and Carsten Wiecher. 2023. Automatic creation of acceptance tests by extracting conditionals from requirements: NLP approach and case study. Journal of Systems and Software 197 (2023), 111549. DOI: 10.1016/j.jss.2022.111549
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76, 5 (1971), 378.
Zhaoqiang Guo, Tingting Tan, Shiran Liu, Xutong Liu, Wei Lai, Yibiao Yang, Yanhui Li, Lin Chen, Wei Dong, and Yuming Zhou. 2023. Mitigating false positive static analysis warnings: Progress, challenges, and opportunities. IEEE Transactions on Software Engineering 49, 12 (2023), 5154–5188.
Shifa-e-Zehra Haidry and Tim Miller. 2013. Using Dependency Structures for Prioritization of Functional Test Suites. IEEE Transactions on Software Engineering 39, 2 (2013), 258–275.
Pascal Hitzler, Aaron Eberhart, Monireh Ebrahimi, Md Kamruzzaman Sarker, and Lu Zhou. 2022. Neuro-symbolic approaches in artificial intelligence. National Science Review 9, 6 (06 2022), nwac035. DOI: 10.1093/nsr/nwac035
Xinmeng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, and Edgar Dobriban. 2024. Uncertainty in Language Models: Assessment through Rank-Calibration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 284–312. DOI: 10.18653/v1/2024.emnlp-main.18
ISO. 2021. IEEE/ISO/IEC International Standard for Software and systems engineering–Software testing–Part 3:Test documentation. ISO/IEC/IEEE 29119-3:2021(E) 2021, 29119-3 (2021), 1–98. DOI: 10.1109/IEEESTD.2021.9591577
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. International Conference on Learning Representations (2023), 19 pages. DOI: 10.48550/arXiv.2302.09664
Yihao Li, Pan Liu, Haiyang Wang, Jie Chu, and W Eric Wong. 2025. Evaluating large language models for software testing. Computer Standards & Interfaces 93 (2025), 103942.
B. B. Liubinskyi and M. M. Zvarych. 2026. A Graph-Based Model for Automatic Test Case Generation from Textual Requirements with Hierarchical Coverage. Mathematical Modeling and Computing 13, 1 (2026), 311–320. DOI: 10.23939/mmc2026.01.311
Jordy Navarro and Ronald Ibarra. 2026. Automatic test case generation using natural language processing: A systematic mapping study. Information and
Rongqi Pan, Taher A. Ghaleb, and Lionel C. Briand. 2024. LTM: Scalable and Black-Box Similarity-Based Test Suite Minimization Based on Language Models. IEEE Transactions on Software Engineering 50, 11 (2024), 2890–2907. DOI: 10.1109/TSE.2024.3465143
Rongqi Pan, Feifei Niu, Lionel C. Briand, and Hanyang Hu. 2026. Requirements Coverage-Guided Minimization for Natural Language Test Cases. ACM Transactions on Software Engineering and Methodology (Feb. 2026), 24 pages. DOI: 10.1145/3798164, Just Accepted.
Yun Peng, Shuqing Li, Wenwei Gu, Yichen Li, Wenxuan Wang, Cuiyun Gao, and Michael R. Lyu. 2022. Revisiting, Benchmarking and Exploring API Recommendation: How Far Are We? IEEE Transactions on Software Engineering 49, 4 (2022), 1876–1897. DOI: 10.1109/TSE.2022.3197600
Zi Peng, Tse-Hsun Chen, and Jinqiu Yang. 2022. Revisiting Test Impact Analysis in Continuous Testing From the Perspective of Code Dependencies. IEEE Transactions on Software Engineering 48, 6 (2022), 1979–1993. DOI: 10.1109/TSE.2020.3045914
Roger S Pressman. 2015. Software engineering: a practitioner’s approach.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Asli Celikyilmaz and Tsung-Hsien Wen (Eds.). Association for Computational Linguistics, Online, 101–108. DOI: 10.18653/v1/2020.acl-demos.14
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Kong, China, 3982–3992. DOI: 10.18653/v1/D19-1410
Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence 1, 5 (2019), 206–215.
Serhad Sarica and Jianxi Luo. 2021. Stopwords in technical language processing. PLOS ONE 16, 8 (2021), e0254937. DOI: 10.1371/journal.pone.0254937
Zhengyan Shi, Giuseppe Castellucci, Simone Filice, Saar Kuzi, Elad Kravi, Eugene Agichtein, Oleg Rokhlenko, and Shervin Malmasi. 2025. Ambiguity Detection and Uncertainty Calibration for Question Answering with Large Language Models. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025), Trista Cao, Anubrata Das, Tharindu Kumarage, YixinWan, Satyapriya Krishna, Ninareh Mehrabi, Jwala Dhamala, Anil Ramakrishna, Aram Galystan, Anoop Kumar, Rahul Gupta, and Kai-Wei Chang (Eds.). Association for Computational Linguistics, Albuquerque, New Mexico, 41–55. DOI: 10.18653/v1/2025.trustnlp-main.4
Amjed Tahir, Shawn Rasheed, Jens Dietrich, Negar Hashemi, and Lu Zhang. 2023. Test flakiness’ causes, detection, impact and responses: A multivocal review. Journal of Systems and Software 206 (2023), 111837. DOI: 10.1016/j.jss.2023.111837
Sahar Tahvili, Leo Hatvani, Michael Felderer, Wasif Afzal, and Markus Bohlin. 2019. Automated Functional Dependency Detection Between Test Cases Using Doc2Vec and Clustering. In 2019 IEEE International Conference On Artificial Intelligence Testing (AITest) (2019-04). IEEE, Newark, CA, USA, 19–26. DOI: 10.1109/AITest.2019.00-13
Sahar Tahvili, Leo Hatvani, Michael Felderer, Francisco Gomes de Oliveira Neto, Wasif Afzal, and Robert Feldt. 2025. Comparative analysis of text mining and clustering techniques for assessing functional dependency between manual test cases. Software Quality Journal 33, 24 (2025), 1–36. DOI: 10.1007/s11219-025-09722-7
Guido Tebes, Luis Olsina, Denis Peppino, and Pablo Becker. 2020. TestTDO: A Top-Domain Software Testing Ontology. In Proceedings of the XXIII Iberoamerican Conference on Software Engineering, CIbSE 2020, Claudia P. Ayala, Leonardo Murta, Daniela Soares Cruzes, Eduardo Figueiredo, Carla Silva, Jose Luis de la Vara, Breno de França, Martín Solari, Guilherme Horta Travassos, and Ivan Machado (Eds.). Curran Associates, Curitiba, Paraná, Brazil, 364–377.
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Transactions on Software Engineering 50, 4 (2024), 911–936. DOI: 10.1109/TSE.2024.3368208
Zhenzhen Yang, Rubing Huang, Chenhui Cui, Nan Niu, and Dave Towey. 2025. Requirements-Based Test Generation: A Comprehensive Survey. ACM Trans. Softw. Eng. Methodol. (Oct. 2025). DOI: 10.1145/3771727, Just Accepted.
Dongran Yu, Bo Yang, Dayou Liu, Hui Wang, and Shirui Pan. 2023. A survey on neural-symbolic learning systems. Neural Netw. 166, C (Sept. 2023), 105–126. DOI: 10.1016/j.neunet.2023.06.028
Zeyu Yu, Ning Fang, Song Wang, and Yiling Ling. 2024. Software Testing with Large Language Models: Survey, Landscape, and Vision. IEEE Transactions on Software Engineering 50, 4 (2024), Detailed pagination or article ID. arXiv:2307.07221 [cs.SE] DOI: 10.1109/TSE.2024.3368208
