Do SAST Alerts Become Reproducible Tests? Evaluating Context-Enriched LLMs for Security Verification
Resumo
Context. Static application security testing (SAST) tools report potential vulnerabilities, but their alerts rarely provide enough guidance for practical verification. Goal. This study investigates whether common weakness enumeration (CWE) descriptions, web security testing guide (WSTG) procedures, and local code snippets improve LLM-generated security verification procedures from SAST alerts. Method. We conducted a controlled repeated-measures experiment with eight SonarQube issues from OWASP Juice Shop. For each issue, procedures were generated under three conditions: isolated SAST alert, alert enriched with CWE/WSTG via retrieval-augmented generation (RAG), and alert enriched with CWE/WSTG plus local code. The procedures were evaluated by judges experienced in software testing, SAST, and pentesting. The rubric measured technical correctness, practical reproducibility, structural completeness, procedural clarity, WSTG alignment, and manual adjustments. Results. Contextual enrichment did not produce statistically consistent improvements in technical correctness, practical reproducibility, procedural clarity, or WSTG alignment. Practical reproducibility remained low across all conditions. The condition with local code showed the lowest median number of manual adjustments, but the effect did not remain significant after correction. Conclusion. Adding security documentation and local code to LLM prompts was not sufficient to transform SAST alerts into reproducible verification procedures. LLMs may support initial drafting, but reliable operationalization still requires human review, execution-based validation, and richer runtime context.
Referências
Greta Dolcetti, Vincenzo Arceri, Eleonora Iotti, Sergio Maffeis, Agostino Cortesi, and Enea Zaffanella. 2026. Ajudando LLMs a melhorar a geração de código usando feedback de testes e análise estática. Discov. Artif. Intell. 6, 1 (March 2026).
Xueying Du, Geng Zheng, Kaixin Wang, Yi Zou, Yujia Wang, Wentai Deng, Jiayi Feng, Mingwei Liu, Bihuan Chen, Xin Peng, Tao Ma, and Yiling Lou. 2026. Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG. ACM Transactions on Software Engineering and Methodology (2 2026). DOI: 10.1145/3797277
Rasmus Ingemann Tuffveson Jensen, Vali Tawosi, and Salwa Alamir. 2024. Software Vulnerability and Functionality Assessment using Large Language Models. In Proceedings of the Third ACM/IEEE InternationalWorkshop on NL-based Software Engineering (New York, NY, USA). ACM, 25–28. DOI: 10.1145/3643787.3648036
Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs. Transactions of the Association for Computational Linguistics 12 (11 2024), 1417–1440. DOI: 10.1162/tacl_a_00713
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33 (2020), 9459–9474.
Kaixuan Li, Sen Chen, Lingling Fan, Ruitao Feng, Han Liu, Chengwei Liu, Yang Liu, and Yixiang Chen. 2023. Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (New York, NY, USA). ACM, 921–933. DOI: 10.1145/3611643.3616262
Ziyang Li, Saikat Dutta, and Mayur Naik. 2025. IRIS: LLM-assisted static analysis for detecting security vulnerabilities. In International Conference on Learning Representations, Vol. 2025. 35735–35758.
Stephan Lipp, Sebastian Banescu, and Alexander Pretschner. 2022. An empirical study on the effectiveness of static C code analyzers for vulnerability detection. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (New York, NY, USA). ACM, 544–555. DOI: 10.1145/3533767.3534380
Qiheng Mao, Zhenhao Li, Xing Hu, Kui Liu, Xin Xia, and Jianling Sun. 2025. Towards Explainable Vulnerability Detection With Large Language Models. IEEE Transactions on Software Engineering 51, 10 (2025), 2957–2971. DOI: 10.1109/TSE.2025.3605442
OWASP Foundation 2021. OWASP Top 10:2021. OWASP Foundation. [link]
Rafael Ramires, Sarmad Bashir, Abbas Khan, Mehrdad Saadatmand, and Ibéria Medeiros. [n. d.]. Friends or Foes? Combining Static Analysis Tools and LLMs for Vulnerability Detection. Technical Report.
Cathrin Schachner and Jasmin Wachter. 2026. Can AI Lower the Barrier to Cybersecurity? A Human-Centered Mixed-Methods Study of Novice CTF Learning. arXiv:2602.18172 [cs.CR] [link]
Jiahao Shi and Tianyi Zhang. 2026. RESCUE: Retrieval Augmented Secure Code Generation. In The Fourteenth International Conference on Learning Representations. [link]
Norbert Tihanyi, Yiannis Charalambous, Ridhi Jain, Mohamed Amine Ferrag, and Lucas C. Cordeiro. 2025. A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verification. In 2025 IEEE/ACM International Conference on Automation of Software Test (AST). 136–147. DOI: 10.1109/AST66626.2025.00020
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Transactions on Software Engineering 50, 4 (2024), 911–936. DOI: 10.1109/TSE.2024.3368208
Jiayimei Wang, Tao Ni, Wei-Bin Lee, and Qingchuan Zhao. 2025. A Contemporary Survey of Large Language Model Assisted Program Analysis. arXiv:2502.18474 [cs.SE] [link]
Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2024. Experimentation in Software Engineering. Springer Berlin Heidelberg. DOI: 10.1007/978-3-662-69306-3
Hanxiang Xu, Shenao Wang, Ningke Li, Kailong Wang, Yanjie Zhao, Kai Chen, Ting Yu, Yang Liu, and Haoyu Wang. 2025. Large Language Models for Cyber Security: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. (Sept. 2025). DOI: 10.1145/3769676, Just Accepted.
Mengyao Zhao, Kaixuan Li, Lyuye Zhang, Wenjing Dang, Chenggong Ding, Sen Chen, and Zheli Liu. 2025. A Systematic Study on Generating Web Vulnerability Proof-of-Concepts Using Large Language Models. arXiv:2510.10148 [cs.SE] [link]
Orçun Çetin and Nazlı Bıyıklı. 2026. Evaluating the Efficacy of Large Language Models for Automated Vulnerable Code Generation. Springer, 197–212. DOI: 10.1007/978-3-032-17443-7_12
