Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation

• Francis Luis Santos Vargas UNIPAMPA
• Rodrigo Brandão Mansilha UNIPAMPA https://orcid.org/0000-0002-2083-653X
• Diego Kreutz UNIPAMPA https://orcid.org/0000-0003-0830-0238

DOI: https://doi.org/10.5753/sbseg.2026.29299

Resumo

Cloud misconfiguration remains a leading cause of security incidents, yet whether LLMs and SLMs can generate security-compliant Infrastructure-as-Code is an open question. We benchmark seven models, three closed LLMs (Claude Opus 4, GPT-5.4, Gemini 2.5 Pro) and four open SLMs (Qwen2.5-Coder-14B, WizardCoder-33B, CodeLlama-13B, Magicoder-S-CL-7B), on AWS Terraform generation across 17 scenarios, integrating Checkov and Trivy scanners into a GitLab CI/CD pipeline and evaluating two prompt strategies at three security levels (pass@5). Syntactic validity and security compliance are largely orthogonal properties in LLM-generated IaC, a model that reliably produces well-formed Terraform does not necessarily produce secure Terraform: WizardCoder-33B achieves 77.8% validate rate yet zero Checkov compliance, while Claude Opus 4 reaches 23.1% Checkov and 92.5% Trivy pass rates under detailed security prompting. Consequently, prompt engineering alone is insufficient: automated multi-tool scanning remains a necessary complement to LLM-assisted IaC generation regardless of model family or prompt strategy. All artifacts are publicly available.

Referências

Chen, M. et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Continella, A., Polino, M., Pogliani, M., and Zanero, S. (2018). There’s a hole in that bucket! A large-scale analysis of misconfigured S3 buckets. In Proceedings of the 34th Annual Computer Security Applications Conference (ACSAC), pages 702–711. ACM.

Davidson, S., Sun, L., Bhasker, B., Callot, L., and Deoras, A. (2025). Multi-IaC-Eval: Benchmarking cloud infrastructure as code across multiple formats. arXiv preprint arXiv:2509.05303.

Fang, C., Miao, N., Srivastav, S., Liu, J., Zhang, R., Fang, R., Tsang, R., Nazari, N., Wang, H., and Homayoun, H. (2024). Large language models for code analysis: Do LLMs really do their job? In 33rd USENIX Security Symposium (USENIX Security), pages 829–846. USENIX Association.

Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2023). OPTQ: Accurate post-training quantization for generative pre-trained transformers. In Proceedings of the 11th International Conference on Learning Representations (ICLR).

Jana, P., Davidson, S., Bhasker, B., Kan, A., Deoras, A., and Callot, L. (2026). TerraFormer: Automated infrastructure-as-code with LLMs fine-tuned via policy-guided verifier feedback. In Proceedings of the IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE/ACM.

Kon, P. T., Liu, J., Qiu, Y., Fan, W., He, T., Lin, L., Zhang, H., Park, O. M., Elengikal, G. S., Kang, Y., et al. (2024). IaC-Eval: A code generation benchmark for cloud infrastructure-as-code programs. In Advances in Neural Information Processing Systems, volume 37, pages 134488–134512.

Morris, K. (2020). Infrastructure as Code: Dynamic Systems for the Cloud Age. O’Reilly Media, 2nd edition.

Nekrasov, R., Fossati, S., Kumara, I., Tamburri, D. A., and van den Heuvel, W.-J. (2026). IaC generation with LLMs: An error taxonomy and a study on configuration knowledge injection. ACM Transactions on Software Engineering and Methodology.

Opdebeeck, R., Zerouali, A., and De Roover, C. (2023). Control and data flow in security smell detection for infrastructure as code: Is it worth the effort? In Proceedings of the 20th IEEE/ACM International Conference on Mining Software Repositories (MSR), pages 534–545. IEEE.

Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., and Karri, R. (2022). Asleep at the keyboard? Assessing the security of GitHub Copilot’s code contributions. In Proceedings of the 43rd IEEE Symposium on Security and Privacy (S&P), pages 754–768. IEEE.

Perry, N., Srivastava, M., Kumar, D., and Boneh, D. (2023). Do users write more insecure code with AI assistants? In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2785–2799. ACM.

Rahman, A., Parnin, C., and Williams, L. (2019). The seven sins: Security smells in infrastructure as code scripts. In Proceedings of the 41st IEEE/ACM International Conference on Software Engineering (ICSE), pages 164–175. IEEE. Distinguished Paper Award.

Rahman, A., Rahman, M. R., Parnin, C., and Williams, L. (2021). Security smells in Ansible and Chef scripts: A replication study. ACM Transactions on Software Engineering and Methodology, 30(1).

Rahman, A., Shamim, S. I., Bose, D. B., and Pandita, R. (2023). Security misconfigurations in open source Kubernetes manifests: An empirical study. ACM Transactions on Software Engineering and Methodology, 32(4).

Saavedra, N. and Ferreira, J. F. (2022). GLITCH: Automated polyglot security smell detection in infrastructure as code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1–12. ACM.

SentinelOne (2024). Cloud security report 2024. Technical report, SentinelOne.

Verdet, A., Hamdaqa, M., Da Silva, L., and Khomh, F. (2025). Assessing the adoption of security policies by developers in Terraform across different cloud providers. Empirical Software Engineering, 30(3).

Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212.

Zhang, T., Pan, S., Zhang, Z., Xing, Z., and Sun, X. (2026). Deployability-centric infrastructure-as-code generation: Fail, learn, refine, and succeed through LLM-empowered DevOps simulation. Proceedings of the ACM on Software Engineering, 3(FSE):321–343.