Aplicando Ferramentas SAST com IA Gratuitas em uma Ferramenta de Ensino de Gestão de Projetos - Um Benchmarking
Resumo
O cenário de segurança de software exige que desenvolvedores integrem ferramentas de Análise Estática de Segurança de Aplicações (SAST) em seus processos de integração contínua (CI/CD). Com a ascensão da Inteligência Artificial (IA) nesse domínio, torna-se crítica a avaliação da acurácia e da usabilidade prática dessas soluções. Este trabalho propõe um benchmarking de validação comparativa para quantificar a eficácia e utilidade de ferramentas SAST freemium com capacidades de IA. A metodologia emprega um Strong-Sense Benchmark (SSB) de mais de 150 mil linhas de código em TypeScript e o Framework Híbrido de Avaliação de SAST (FHA-SAST). O FHA-SAST combina princípios de Verificação e Validação (V&V), métricas de precisão (Verdadeiros/Falsos Positivos) e modelos de aceitação tecnológica (TAM), normalizando todos os resultados a uma escala Likert de 1 a 5. Os resultados demonstram um trade-off evidente entre ferramentas: o Semgrep alcançou a maior precisão (96,88%), indicando o menor ruído, porém com o maior tempo de execução. Em contraste, o SonarCloud obteve a melhor utilidade percebida (5.0), mas com menor precisão objetiva. Concluímos que a adoção ideal de ferramentas SAST/IA exige a análise integrada de desempenho técnico, precisão e fatores logísticos de usabilidade. O FHA-SAST é validado como um método eficaz para guiar a seleção de ferramentas em ambientes de desenvolvimento modernos.
Palavras-chave:
Análise Estática de Segurança de Aplicações, Inteligência Artificial, Benchmarking de Ferramentas, Segurança de Software, SAST, DevSecOps
Referências
Alan Agresti. 2010. Analysis of ordinal categorical data. Vol. 656. John Wiley & Sons.
Nikolaos Alexopoulos, Manuel Brack, Jan Philipp Wagner, Tim Grube, and Max Mühlhäuser. 2022. How Long Do Vulnerabilities Live in the Code? A Large-Scale Empirical Measurement Study on FOSS Vulnerability Lifetimes. In 31st USENIX Security Symposium (USENIX Security 22). USENIX Association, Boston, MA, 359–376. [link]
Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR) 51, 4 (2018), 1–37.
Bandar Aloraini, Massimiliano Menarini, and William G. Griswold. 2019. An Empirical Study of Security Warnings from Static Analysis Tools. Journal of Systems and Software 158 (2019), 110414. DOI: 10.1016/j.jss.2019.110414
Gabor Antal and Sebastian Banescu. 2021. Challenges in benchmarking static analysis tools. In Proceedings of the 2021 IEEE/ACM 1st International Conference on AI Engineering.
Farid Binbeshr and Muhammad Imam. 2025. Comparative Analysis of AI-Driven Security Approaches in DevSecOps: Challenges, Solutions, and Future Directions. arXiv preprint arXiv:2504.19154 (2025). DOI: 10.48550/arXiv.2504.19154
Valentina Casola, Alessandra De Benedictis, Carmine Mazzocca, and Vincenzo Orbinato. 2024. Secure Software Development and Testing: A Model-Based Approach Integrating Safety and Security for Industry 4.0. Computers & Security 137 (2024), 103639. DOI: 10.1016/j.cose.2023.103639
Brian Chess and Jacob West. 2007. Secure Programming with Static Analysis. Addison-Wesley Professional, Boston.
Roland Croft, Dominic Newlands, Ziyu Chen, and M. Ali Babar. 2021. An Empirical Study of Rule-Based and Learning-Based Approaches for Static Application Security Testing. In Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1–12. DOI: 10.1145/3475716.3475781
Fred D Davis. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly (1989), 319–340.
AR Gray and RB Rainer Jr. 2003. Quantitative and qualitative analysis of factors affecting software process. Empirical Software Engineering 8 (2003), 293–306.
Emanuele Iannone, Roberta Guadagni, Filomena Ferrucci, Andrea De Lucia, and Fabio Palomba. 2023. The Secret Life of Software Vulnerabilities: A Large-Scale Empirical Study. IEEE Transactions on Software Engineering 49, 1 (2023), 44–63. DOI: 10.1109/TSE.2022.3140868
Frank Li et al. 2025. Understanding Industry Perspectives of Static Application Security Testing (SAST) Evaluation. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1–12.
Kaixuan Li, Sen Chen, Yue Fang, et al. 2023. Comparison and Evaluation on Static Application Security Testing Tools. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 596–608.
Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, and Yawei Zhu. 2018. VulDeePecker: A deep learning-based system for vulnerability detection. In Proceedings of the Network and Distributed System Security Symposium (NDSS).
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Vol. 1. Cambridge University Press.
Robert C Martin. 2017. Clean Architecture: A Craftsman’s Guide to Software Structure and Design. Prentice Hall.
Meta Platforms, Inc. 2020. React - A JavaScript library for building user interfaces. [link]. Acessado em Dez. 2025.
Microsoft Corporation. 2020. TypeScript Language. [link]. Acessado em Dez. 2025.
Dionas Luan Müller, Matheus Lemes Fialho, Elder de Macedo Rodrigues, and Maicon Bernardino. 2026. Aplicando Ferramentas SAST com IA Gratuitas em uma Ferramenta de Ensino de Gestão de Projetos - Um Benchmarking - Dataset. DOI: 10.5281/zenodo.21866721.
Node.js Foundation. 2020. Node.js - JavaScript runtime built on Chrome’s V8 JavaScript engine. [link]. Acessado em Dez. 2025.
P. Nunes, I. Medeiros, J. Fonseca, and M. Vieira. 2018. Benchmarking static analysis tools for web security. In 2018 IEEE 17th International Symposium on Network Computing and Applications (NCA). IEEE, 1–8.
William L Oberkampf and Timothy Guy Trucano. 2006. Design of and Comparison with Verification and Validation Benchmarks. Technical Report. Sandia National Lab.(SNL-NM), Albuquerque, NM (United States).
Omitted et al. 2025. Omitted. arXiv preprint arXiv:omitted (2025).
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? Assessing the security of GitHub Copilot’s generated code. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 754–768.
PMI. 2021. A Guide to the Project Management Body of Knowledge (PMBOK Guide) (7th ed.). Project Management Institute, Newtown Square, PA, USA.
Riccardo Scandariato, JamesWalden, Aram Hovsepyan, andWouter Joosen. 2014. Predicting vulnerable software components via text mining. IEEE Transactions on Software Engineering 40, 10 (2014), 993–1006.
Gabriel Sousa and Rodrigo Bonifácio. 2024. Mapeamento Sistemático de Comparativos de Ferramentas de Análise Estática de Segurança de Aplicações. In Anais Estendidos do XXIV Simpósio Brasileiro de Segurança da Informação e de Sistemas Computacionais. 281–288.
Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. 2023. An empirical study of deep learning models for vulnerability detection. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2237–2248.
Stanley Smith Stevens. 1946. On the theory of scales of measurement. Science 103, 2684 (1946), 677–680.
Gail Sullivan and Anthony Artino. 2013. Analyzing and Interpreting Data From Likert-Type Scales. Journal of graduate medical education 5 (12 2013), 541–2. DOI: 10.4300/JGME-5-4-18
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
Xiaofan Zhao, Tony Clear, and Ramesh Lal. 2024. Identifying the Primary Dimensions of DevSecOps: A Multi-vocal Literature Review. Journal of Systems and Software 214 (2024), 112063. DOI: 10.1016/j.jss.2024.112063
Nikolaos Alexopoulos, Manuel Brack, Jan Philipp Wagner, Tim Grube, and Max Mühlhäuser. 2022. How Long Do Vulnerabilities Live in the Code? A Large-Scale Empirical Measurement Study on FOSS Vulnerability Lifetimes. In 31st USENIX Security Symposium (USENIX Security 22). USENIX Association, Boston, MA, 359–376. [link]
Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR) 51, 4 (2018), 1–37.
Bandar Aloraini, Massimiliano Menarini, and William G. Griswold. 2019. An Empirical Study of Security Warnings from Static Analysis Tools. Journal of Systems and Software 158 (2019), 110414. DOI: 10.1016/j.jss.2019.110414
Gabor Antal and Sebastian Banescu. 2021. Challenges in benchmarking static analysis tools. In Proceedings of the 2021 IEEE/ACM 1st International Conference on AI Engineering.
Farid Binbeshr and Muhammad Imam. 2025. Comparative Analysis of AI-Driven Security Approaches in DevSecOps: Challenges, Solutions, and Future Directions. arXiv preprint arXiv:2504.19154 (2025). DOI: 10.48550/arXiv.2504.19154
Valentina Casola, Alessandra De Benedictis, Carmine Mazzocca, and Vincenzo Orbinato. 2024. Secure Software Development and Testing: A Model-Based Approach Integrating Safety and Security for Industry 4.0. Computers & Security 137 (2024), 103639. DOI: 10.1016/j.cose.2023.103639
Brian Chess and Jacob West. 2007. Secure Programming with Static Analysis. Addison-Wesley Professional, Boston.
Roland Croft, Dominic Newlands, Ziyu Chen, and M. Ali Babar. 2021. An Empirical Study of Rule-Based and Learning-Based Approaches for Static Application Security Testing. In Proceedings of the 15th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1–12. DOI: 10.1145/3475716.3475781
Fred D Davis. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly (1989), 319–340.
AR Gray and RB Rainer Jr. 2003. Quantitative and qualitative analysis of factors affecting software process. Empirical Software Engineering 8 (2003), 293–306.
Emanuele Iannone, Roberta Guadagni, Filomena Ferrucci, Andrea De Lucia, and Fabio Palomba. 2023. The Secret Life of Software Vulnerabilities: A Large-Scale Empirical Study. IEEE Transactions on Software Engineering 49, 1 (2023), 44–63. DOI: 10.1109/TSE.2022.3140868
Frank Li et al. 2025. Understanding Industry Perspectives of Static Application Security Testing (SAST) Evaluation. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1–12.
Kaixuan Li, Sen Chen, Yue Fang, et al. 2023. Comparison and Evaluation on Static Application Security Testing Tools. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 596–608.
Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, and Yawei Zhu. 2018. VulDeePecker: A deep learning-based system for vulnerability detection. In Proceedings of the Network and Distributed System Security Symposium (NDSS).
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Vol. 1. Cambridge University Press.
Robert C Martin. 2017. Clean Architecture: A Craftsman’s Guide to Software Structure and Design. Prentice Hall.
Meta Platforms, Inc. 2020. React - A JavaScript library for building user interfaces. [link]. Acessado em Dez. 2025.
Microsoft Corporation. 2020. TypeScript Language. [link]. Acessado em Dez. 2025.
Dionas Luan Müller, Matheus Lemes Fialho, Elder de Macedo Rodrigues, and Maicon Bernardino. 2026. Aplicando Ferramentas SAST com IA Gratuitas em uma Ferramenta de Ensino de Gestão de Projetos - Um Benchmarking - Dataset. DOI: 10.5281/zenodo.21866721.
Node.js Foundation. 2020. Node.js - JavaScript runtime built on Chrome’s V8 JavaScript engine. [link]. Acessado em Dez. 2025.
P. Nunes, I. Medeiros, J. Fonseca, and M. Vieira. 2018. Benchmarking static analysis tools for web security. In 2018 IEEE 17th International Symposium on Network Computing and Applications (NCA). IEEE, 1–8.
William L Oberkampf and Timothy Guy Trucano. 2006. Design of and Comparison with Verification and Validation Benchmarks. Technical Report. Sandia National Lab.(SNL-NM), Albuquerque, NM (United States).
Omitted et al. 2025. Omitted. arXiv preprint arXiv:omitted (2025).
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? Assessing the security of GitHub Copilot’s generated code. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 754–768.
PMI. 2021. A Guide to the Project Management Body of Knowledge (PMBOK Guide) (7th ed.). Project Management Institute, Newtown Square, PA, USA.
Riccardo Scandariato, JamesWalden, Aram Hovsepyan, andWouter Joosen. 2014. Predicting vulnerable software components via text mining. IEEE Transactions on Software Engineering 40, 10 (2014), 993–1006.
Gabriel Sousa and Rodrigo Bonifácio. 2024. Mapeamento Sistemático de Comparativos de Ferramentas de Análise Estática de Segurança de Aplicações. In Anais Estendidos do XXIV Simpósio Brasileiro de Segurança da Informação e de Sistemas Computacionais. 281–288.
Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. 2023. An empirical study of deep learning models for vulnerability detection. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2237–2248.
Stanley Smith Stevens. 1946. On the theory of scales of measurement. Science 103, 2684 (1946), 677–680.
Gail Sullivan and Anthony Artino. 2013. Analyzing and Interpreting Data From Likert-Type Scales. Journal of graduate medical education 5 (12 2013), 541–2. DOI: 10.4300/JGME-5-4-18
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
Xiaofan Zhao, Tony Clear, and Ramesh Lal. 2024. Identifying the Primary Dimensions of DevSecOps: A Multi-vocal Literature Review. Journal of Systems and Software 214 (2024), 112063. DOI: 10.1016/j.jss.2024.112063
Publicado
08/09/2026
Como Citar
MÜLLER, Dionas Luan; LEMES FIALHO, Matheus; RODRIGUES, Elder de Macedo; BERNARDINO, Maicon.
Aplicando Ferramentas SAST com IA Gratuitas em uma Ferramenta de Ensino de Gestão de Projetos - Um Benchmarking. In: SIMPÓSIO BRASILEIRO DE TESTES DE SOFTWARE SISTEMÁTICO E AUTOMATIZADO (SAST), 11. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 129-138.
DOI: https://doi.org/10.5753/sast.2026.31222.
