Large Language Models in Software Testing: A Scoping Review

• Ruby Marshall UFLA
• Maurício Magri Ribas das Neves UFLA
• Bento Rafael Siqueira UFLA
• Samira Santos da Silva UFLA

DOI: https://doi.org/10.5753/vem.2026.33027

Resumo

Large Language Models (LLMs) have rapidly emerged as promising tools for supporting Software Engineering activities, particularly Software Testing, by assisting in tasks such as test generation, specification creation, and test data production. Despite the growing number of studies in this area, the current landscape of LLM applications, challenges, and research opportunities remains fragmented. This Scoping Reviewmaps the use of LLMs across different Software Testing activities. A literature search was conducted in the IEEE Xplore and ACM Digital Library databases, covering publications from 2020 to 2025. The search retrieved 1,378 records; after removing 57 duplicates, 1,321 unique studies were screened, resulting in 12 primary studies included in the final synthesis. The findings indicate that LLMs are predominantly applied to the generation of testing artifacts, including unit tests, test inputs, formal specifications, and test oracles. The most frequently reported challenges involve reliability and hallucinations, variability across models and prompts, context and scalability limitations, and detection accuracy. The findings also indicate a trend toward hybrid testing workflows that integrate LLMs with traditional Verification and Validation (V&V) techniques. Overall, the evidence suggests that LLMs have potential to enhance software testing activities when employed as complementary components within reliable and reproducible testing processes.

Palavras-chave: Large Language Models, Software Testing, Scoping Review

Referências

Paul Ammann and Jeff Offutt. 2017. Introduction to Software Testing (2 ed.). Cambridge University Press.

Hilary Arksey and Lisa O’Malley. 2005. Scoping Studies: Towards a Methodological Framework. International Journal of Social Research Methodology 8, 1 (2005), 19–32. DOI: 10.1080/1364557032000119616

Mohamed Boukhlif, Nassim Kharmoum, and Mohamed Hanine. 2024. Llms for intelligent software testing: a comparative study. In Proceedings of the 7th International Conference on Networking, Intelligent Systems and Security. 1–8.

Angela Fan et al. 2024. A Survey of Large Language Models for Software Engineering. arXiv preprint arXiv:2402.06196 (2024).

Z. Fang, J. Li, A. Liang, G. R. Bai, and Y. Huang. 2025. A Comparative Study on ChatGPT and Checklist as Support Tools for Unit Testing Education. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering (FSE Companion). ACM, 871–882.

M. Görmez, B. Turhan, A. Bener, and A. Camkiran. 2024. Large Language Models for Software Engineering: A Systematic Mapping Study. In EuroSPI 2024 (Communications in Computer and Information Science). Springer.

X. et al. Hou. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024).

Y. Huang, J. Wang, C. Chen, Z. Liu, S. Wang, and Q. Wang. 2024. Software Testing with Large Language Models: Survey, Landscape, and Vision. IEEE Transactions on Software Engineering 50, 4 (2024), 911–936.

Ziwei Ji et al. 2023. Survey of Hallucination in Natural Language Generation. Comput. Surveys 55, 12 (2023), 1–38.

M. Kim, T. Stennett, S. Sinha, and A. Orso. 2025. A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1409–1421.

U. Kulsum, H. Zhu, B. Xu, and M. d’Amorim. 2024. A Case Study of LLM for Automated Vulnerability Repair: Assessing Impact of Reasoning and Patch Validation Feedback. In Proceedings of the 1st ACM International Conference on AI-Powered Software (AIware). ACM, 103–111.

J. et al. Li. 2025. LLM Based Input Space Partitioning Testing for Library APIs. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1436–1448.

L. Ma, S. Liu, Y. Li, X. Xie, and L. Bu. 2025. SpecGen: Automated Generation of Formal Program Specifications via Large Language Models. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 16–28.

L. Margabandu, Z. Chen, W. E. Wong, and C.-W. Hsu. 2025. A Design of Experiments Oracle LLM for Research-Grounded Combinatorial Testing. In 2025 25th International Conference on Software Quality, Reliability, and Security Companion (QRS-C). IEEE, 82–90.

Glenford J. Myers, Corey Sandler, and Tom Badgett. 2011. The Art of Software Testing (3 ed.). John Wiley & Sons.

Z. Nan, Z. Guo, K. Liu, and X. Xia. 2025. Test Intention Guided LLM-Based Unit Test Generation. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1026–1038.

Michaela D. B. Peters, Christina M. Godfrey, Hanan Khalil, Patricia McInerney, Deborah Parker, and Cassia Baldini Soares. 2020. Guidance for Conducting Systematic Scoping Reviews. JBI Evidence Synthesis 18, 10 (2020), 2119–2126. DOI: 10.11124/JBIES-20-00167

F. Qi, Y. Hou, N. Lin, S. Bao, andN. Xu. 2024. A Survey of Testing Techniques Based on Large Language Models. In Proceedings of the 2024 International Conference on Computer and Multimedia Technology (ICCMT). ACM, 280–284.

J. Rawson and S. Reddivari. 2025. A ChatGPT-Powered Tool for Automating Context-Aware Acceptance Criteria Generation for User Stories. In 2025 IEEE International Conference on Information Reuse and Integration and Data Science (IRI). IEEE, 325–330.

Marlen Treviğno-Villalobos, C. Quesada-López, E. Jiménez-Delgado, R. QuirósOviedo, and I. Díaz-Oreiro. 2025. A Comparative Evaluation of ChatGPT, DeepSeek and Gemini in Automatic Unit Test Generation: a Success Rate Analysis. In 2025 IEEE VIII Congreso Internacional en Inteligencia Ambiental, Ingenieria de Software y Salud Electronica y Movil (AmITIC). IEEE, 1–8.

Andrea C. Tricco, Erin Lillie, Wasifa Zarin, Kelly K. O’Brien, Heather Colquhoun, Danielle Levac, David Moher, et al. 2018. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine 169, 7 (2018), 467–473. DOI: 10.7326/M18-0850

W. Wei. 2025. Static Analysis and LLM for Comprehensive Java Unit Test Generation. In 2025 8th International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE). IEEE, 87–92.

J. et al. Yu. 2025. A Preliminary Study of Large Language Models for Multilingual Vulnerability Detection. In Proceedings of the 34th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA Companion). ACM, 161–168.

S. Zhou, J. Wang, H. Ye, H. Zhou, C. L. Goues, and X. Luo. 2025. LWDIFF: an LLM-Assisted Differential Testing Framework for Webassembly Runtimes. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 153–164.