Metadata Is Not Enough: Characterizing Pre-Execution Issues in Mixtral-Generated Kubernetes Validation Commands
Resumo
Cloud-native deployments evolve through manifests and operational checks that must remain synchronized with declared resources and runtime assumptions. We report an exploratory preexecution audit of 34 validation files and 178 step-level CLI command units produced with Mixtral-8x7B-Instruct-v0.1 for 10 evolution comparisons of a Kubernetes-based financial application. The corpus is descriptive, single-model evidence and does not include a quota-balanced generic-prompt baseline. Among all command units, 138 (77.5%) referenced at least one declared Kubernetes resource; the complementary group includes application-level commands for which such a reference is not necessarily expected. Moreover, 136 commands (76.4%) used kubectl. Eight commands (4.5%) retained unresolved placeholders, and 11 (6.2%) referred to undeclared resources, with 9 of those 11 occurrences concentrated in one comparison. A further 143 commands (80.3%) explicitly encoded -n default; because this namespace is correct for the studied manifests, we treat it as a portability-related binding rather than an error. These selected textual properties show that manifest-focused commands can coexist with unresolved or undeclared references. Because the commands were not executed, the study does not assess executability, validation effectiveness, or fault detection. The findings therefore frame metadata as a focusing signal—not as evidence that selected pre-execution issues are absent—and motivate runtime-aware grounding, structured command schemas, and post-generation checks for CI/CD and GitOps workflows.
Referências
Lianping Chen. 2015. Continuous Delivery: Huge Benefits, but Challenges Too. IEEE Software 32, 2 (2015), 50–54. DOI: 10.1109/MS.2015.27
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, Melbourne, Australia, 31–53. DOI: 10.1109/ICSE-FoSE59343.2023.00008
Michele Guerriero, Martin Garriga, Damian A. Tamburri, and Fabio Palomba. 2019. Adoption, Support, and Challenges of Infrastructure-as-Code: Insights from Industry. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, Cleveland, OH, USA, 580–589. DOI: 10.1109/ICSME.2019.00092
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology 33, 8, Article 220 (2024), 79 pages. DOI: 10.1145/3695988
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiannan Yu, Dan Su, Yan Xu, Christy Dougherty, Justin S. Y. Fu, and Maosong Sun. 2023. Survey of Hallucination in Natural Language Generation. Comput. Surveys 55, 12 (2023), 1–38. DOI: 10.1145/3571730
Patrick Tser Jern Kon, Jiachen Liu, Yiming Qiu, Weijun Fan, Ting He, Lei Lin, Haoran Zhang, Owen M. Park, George Sajan Elengikal, Yuxin Kang, Ang Chen, Mosharaf Chowdhury, Myungjin Lee, and Xinyu Wang. 2024. IaC-Eval: A Code Generation Benchmark for Cloud Infrastructure-as-Code Programs. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, Datasets and Benchmarks Track. Curran Associates, Inc., Vancouver, Canada, 134488–134506. [link]
Marko Leppanen, Simo Makinen, Max Pagels, Vesa-Pekka Eloranta, Juha Itkonen, Mika Mantyla, and Tomi Mannisto. 2015. The Highways and Country Roads to Continuous Deployment. IEEE Software 32, 2 (2015), 64–72. DOI: 10.1109/MS.2015.50
Kief Morris. 2016. Infrastructure as Code: Managing Servers in the Cloud. O’Reilly Media, Inc., Sebastopol, CA. [link]
Claus Pahl, Antonio Brogi, Jacopo Soldani, and Pooyan Jamshidi. 2017. Cloud Container Technologies: A State-of-the-Art Review. IEEE Transactions on Cloud Computing 7, 3 (2017), 677–692. DOI: 10.1109/TCC.2017.2702586
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In Proceedings of the IEEE Symposium on Security and Privacy. IEEE, San Francisco, CA, 754–768. DOI: 10.1109/SP46214.2022.9833571
Akond Rahman and Laurie Williams. 2018. Characterizing Defective Configuration Scripts Used for Continuous Deployment. In Proceedings of the 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). IEEE, Västerås, Sweden, 34–45. DOI: 10.1109/ICST.2018.00014
Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. arXiv:2102.07350 [cs.CL] [link]
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing with Large Language Models: Survey, Landscape, and Vision. IEEE Transactions on Software Engineering 50, 4 (2024), 911–936. DOI: 10.1109/TSE.2024.3368208
JulesWhite, Quchen Fu, SamHays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashis Ghosh, Andrew C. Cates, Sydney Heaton, Eric Gombolay, Jeff Jenkins, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 [cs.SE] [link]
Yifei Xu, Yuning Chen, Xumiao Zhang, Xianshang Lin, Pan Hu, Yunfei Ma, Songwu Lu, Wan Du, Zhuoqing Mao, Ennan Zhai, and Dennis Cai. 2024. CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generation. arXiv:2401.06786 [cs.SE] [link]
J. D. Zamfirescu-Pereira, Richmond Wong, Benedetta Piantella, and Qian Yang. 2023. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1, Article 123 (2023), 32 pages. DOI: 10.1145/3544548.3581388
