Comparative evaluation of the performance of generative artificial intelligences and traditional tools in JavaScript source code analysis

• Rayane Pimentel Senac
• Claudia Bianchi Progetti Senac

DOI: https://doi.org/10.5753/sbseg_estendido.2025.11655

Abstract

Comparative study between SAST tools (Semgrep/SonarQube) and LLM models (DeepSeek/CodeLlama) for detecting JavaScript vulnerabilities (OWASP Juice Shop). Results reveal complementarity: SASTs achieve 100% precision for standard vulnerabilities (XSS/SQLi), while LLMs offer higher recall (70% in DeepSeek) for contextual threats (NoSQLi/access control). The 2245% false positive rate in LLMs calls for filtering strategies. We demonstrate that hybrid pipelines optimally combine SAST accuracy with LLM coverage. The study contributes empirical evidence for the adoption of LLMs in security pipelines, highlighting challenges such as the mitigation of false positives.

References

Hu, T. et al. (2024). Unveiling llm evaluation focused on metrics: Challenges and solutions. arXiv. Disponível em: [link]. Acesso em: 05 abr. 2025.

IBM (2024). Análise de código de ia. Disponível em: [link]. Acesso em: 05 abr. 2025.

International Organization for Standardization (2023). ISO/IEC 25010:2023 – Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models. ISO, Genebra.

Kiminich, B. (2024). Owasp juice shop: An intentionally insecure javascript web application. Versão 17.3.0. Disponível em: [link]. Acesso em: 20 fev. 2024.

Le, T. K. et al. (2024). A study of vulnerability repair in javascript programs with large language models. In Companion Proceedings of the ACM Web Conference 2024. Disponível em: [link]. Acesso em: 19 mar. 2025.

OWASP Foundation (2023). Owasp top 10:2023 – the ten most critical web application security risks. Disponível em: [link]. Acesso em: 20 fev. 2024.

Palo Alto Networks. What is static application security testing (sast)? Disponível em: [link]. Acesso em: 05 abr. 2025.

Snyk (2022). Snyk top 10 – inteligência de vulnerabilidades de segurança. Disponível em: [link]. Acesso em: 20 fev. 2024.

Stack Overflow. Developer survey 2023. Disponível em: [link]. Acesso em: 20 fev. 2025.

W3Techs (2024). Usage statistics of javascript as client-side programming language on websites. Disponível em: [link]. Acesso em: 20 fev. 2024.

Wadhams, N. et al. (2024). Barriers to using static application security testing (sast) tools: A literature review. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering Workshops, pages 161–166, Sacramento, CA, USA. Association for Computing Machinery. Disponível em: [link]. Acesso em: 19 mar. 2025.

Zhou, X. et al. (2024). Comparison of static application security testing tools and large language models for repo-level vulnerability detection. arXiv. Disponível em: [link]. Acesso em: 15 nov. 2024.