AutoMutSec: Mutation Testing for Authorization Policies in Spring Security

• Caio Jhonatan Alves Pereira UFCG
• Everton L. G. Alves UFCG

DOI: https://doi.org/10.5753/sast.2026.31163

Resumo

Authorization failures remain one of the most critical security risks in modern web applications, yet the effectiveness of authorization testing is rarely assessed. In Spring Security, authorization policies are implemented through annotations and configuration constructs that fall outside the scope of conventional mutation operators, which primarily target language-level faults. As a result, test suites can achieve high mutation scores, while access-control behavior remains largely untested. This paper presents AutoMutSec, a mutation testing tool that introduces ten authorization-specific mutation operators for Spring Security applications. We evaluated AutoMutSec on five real-world Spring Boot systems and compared its results with those produced by PIT, the standard mutation testing tool for Java. AutoMutSec generated 1,745 mutants across the evaluated systems, yielding mutation scores ranging from 0% to 82.42%. We found no consistent relationship between AutoMutSec scores and traditional testing metrics such as PIT mutation scores or code coverage. Notably, some systems with PIT scores above 50% achieved AutoMutSec scores of 0%, revealing that authorization logic can remain entirely untested despite otherwise strong testing indicators. These results suggest that authorization testing effectiveness and general testing effectiveness may be largely independent dimensions. The additional execution overhead remained practical in most systems. Overall, our findings indicate that traditional mutation testing is insufficient to assess authorization testing adequacy and that authorization-specific mutation operators provide complementary insights into security testing effectiveness.

Palavras-chave: Mutation Testing, Spring Security, Authorization Testing, Security Testing

Referências

Edrian S. Abduhari, Tadzher C. Shaik, Alsimar B. Adidul, Jimrashier H. Ladja, Ersin S. Saliddin, Akshay J. Adin, Fradzkhan A. Rumbahali, and Alnadzri B. Sali. 2024. Access Control Mechanisms and Their Role in Preventing Unauthorized Data Access: A Comparative Analysis of RBAC, MFA, and Strong Passwords. Natural Sciences Engineering and Technology Journal (2024). DOI: 10.37275/nasetjournal.v5i1.62

Abdurrasyid, Yudistira Asnar, and Gusti Ayu Putri Saptawati. 2026. Mutation based improvement of security test case quality for broken access control. Journal on Information Security (2026).

Richard Baker and Ibrahim Habli. 2012. An Empirical Evaluation of Mutation Testing for Improving the Test Quality of Safety-Critical Software. (2012).

Brasil. 2018. Lei Geral de Proteção de Dados Pessoais (Lei nº 13.709, de 14 de agosto de 2018). [link]. Accessed: June 8, 2026.

Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: A Practical Mutation Testing Tool for Java. In Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA). ACM, 449–452. DOI: 10.1145/2931037.2948707

European Parliament and Council of the European Union. 2016. Regulation (EU) 2016/679 (General Data Protection Regulation). [link]. Accessed: June 8, 2026.

Andreas Happe and Jürgen Cito. 2023. Understanding hackers’ work: An empirical study of offensive security practitioners. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1669–1680.

Mazharul Islam, Sazzadur Rahaman, Na Meng, Behnaz Hassanshahi, Padmanabhan Krishnan, and Danfeng Daphne Yao. 2020. Coding Practices and Recommendations of Spring Security for Enterprise Applications. In 2020 IEEE Secure Development (SecDev). 49–57. DOI: 10.1109/SecDev45635.2020.00024

Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. DOI: 10.1109/TSE.2010.62

M. Mythily, A. Samson Arun Raj, and Iwin Thanakumar Joseph. 2022. An Analysis of the Significance of Spring Boot in The Market. In 2022 International Conference on Inventive Computation Technologies (ICICT). IEEE. DOI: 10.1109/ICICT54344.2022.9850910

B. Potter and G. McGraw. 2004. Software security testing. IEEE Security Privacy 2, 5 (2004), 81–85. DOI: 10.1109/MSP.2004.84

Ana B. Sánchez, José A. Parejo, Sergio Segura, Amador Durán, and Mike Papadakis. 2024. Mutation Testing in Practice: Insights from Open-Source Software Developers. IEEE Transactions on Software Engineering (2024). DOI: 10.1109/TSE.2024.3377378

Hossain Shahriar and Mohammad Zulkernine. 2008. MUSIC: Mutation-based SQL injection vulnerability checking. In 2008 The Eighth International Conference on Quality Software. IEEE, 77–86.

Yves Le Traon, Tejeddine Mouelhi, and Benoit Baudry. 2007. Testing Security Policies: Going Beyond Functional Testing. In The 18th IEEE International Symposium on Software Reliability (ISSRE ’07). 93–102. DOI: 10.1109/ISSRE.2007.27

Li Zhong. 2023. A Survey of Prevent and Detect Access Control Vulnerabilities. arXiv abs/2304.10600 (2023). DOI: 10.48550/arXiv.2304.10600, Accessed: June 8, 2026.