Configuring Parameters of Search-based Software Engineering Tools with Multiple LLM Models

• Daniel Nouchi UEM
• Willian M. Freire UEM
• Aline M. M. M. Amaral UEM
• Silvia R. Vergilio UFPR
• Thelma E. Colanzi UEM

DOI: https://doi.org/10.5753/sbes.2025.11005

Resumo

Search-Based Software Engineering (SBSE) tools offer automatic support by applying search-based techniques to solve different optimization problems of various software engineering areas. However, configuring SBSE tools often requires expertise in optimization techniques, posing a significant barrier for non-specialist users. This task involves selecting suitable algorithms, tuning parameters, choosing appropriate objective functions and search operators, and making decisions that affect optimization quality and performance. To address this challenge, we propose a multiple-LLM approach that integrates Large Language Models (LLMs), specifically ChatGPT-4, into SBSEworkflows to provide context-aware and natural language suggestions for parameter configuration. The approach follows a Divide and Conquer strategy, in which specialized LLMs are trained for distinct parameter configuration subproblems. As a proof of concept, we implemented OPLA-Wizard that works with OPLA-Tool v2.0, for Product Line Architecture (PLA) design. This assistant supports users in configuring algorithm settings, search operators, and objective functions through a user-friendly interface. Preliminary results show that the multiple-LLM approach improves accuracy, explanation clarity, and output format adherence compared to a single-LLM approach. This work contributes to the democratization of SBSE by reducing cognitive load, preventing misconfigurations, and making advanced optimization techniques more accessible to a broader audience.

Palavras-chave: Search-based Software Engineering, Large Language Model, SPL

Referências

Baleegh Ahmad, Shailja Thakur, Benjamin Tan, Ramesh Karri, and Hammond Pearce. 2024. On Hardware Security Bug Code Fixes By Prompting Large Language Models. IEEE Transactions on Information Forensics and Security (2024). DOI: 10.1109/TIFS.2024.3374558

Antonis Antoniades, Albert Örwall, Kexun Zhang, Yuxi Xie, Anirudh Goyal, and WilliamWang. 2024. SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement. arXiv preprint arXiv:2410.20285 (2024).

Andrea Arcuri and Gordon Fraser. 2013. Parameter tuning or default values? An empirical investigation in search-based software engineering. Empirical Software Engineering 18 (2013), 594–623. DOI: 10.1007/s10664-013-9249-9

Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. 150–158. DOI: 10.48550/arXiv.2309.15217

Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and JieMZhang. 2023. Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53.

Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 416–419. DOI: 10.1145/2025113.2025179

Willian M Freire, Murilo Boccardo, Daniel Nouchi, Aline MMM Amaral, Silvia R Vergilio, Thiago Ferreira, and Thelma E Colanzi. 2024. AIssistDM: A Plugin to Assist Non-specialist Decision-Makers in Search-Based Software Engineering Tools. In Brazilian Symposium on Software Engineering. SBC, 734–740. DOI: 10. 5753/sbes.2024.3567

Willian M Freire, Murilo Boccardo, Daniel Nouchi, Aline MMM Amaral, Silvia R Vergilio, Thiago Ferreira, and Thelma E Colanzi. 2024. Large Language Model-based suggestion of objective functions for search-based Product Line Architecture design. In Simpósio Brasileiro de Componentes, Arquiteturas e Reutilização de Software (SBCARS). SBC, 21–30.

Willian Marques Freire, Mamoru Massago, Arthur Cattaneo Zavadski, Aline Maria Malachini, Miotto Amaral, and Thelma Elita Colanzi. 2020. OPLATool v2. 0: a tool for product line architecture design optimization. In Proceedings of the XXXIV Brazilian Symposium on Software Engineering. 818–823. DOI: 10.1145/3422392.3422498

Mark Harman and Bryan F Jones. 2007. The current state and future of search based software engineering. Future of Software Engineering (2007), 342–357.

William B. Langdon and Mark Harman. 2015. Optimizing Existing Software With Genetic Programming. IEEE Transactions on Evolutionary Computation 19, 1 (2015), 118–135. DOI: 10.1109/TEVC.2013.2281544

Bingdong Li, Zixiang Di, Yanting Yang, Hong Qian, Peng Yang, Hao Hao, Ke Tang, and Aimin Zhou. 2024. It’s Morphing Time: Unleashing the Potential of Multiple LLMs via Multi-objective Optimization. arXiv preprint arXiv:2407.00487 (2024).

Daniel Nouchi, Willian Marques Freire, Aline Maria Malachini Miotto Amaral, Silvia Regina Vergilio, and Thelma Elita Colanzi. 2025. Complementary Material. (2025). DOI: 10.6084/m9.figshare.28930388.v2

Yanyan Peng, Xinjie Wang, Ziyang Chen, Hong Liu, Ge Li, Xin Xia, and Zhi Jin. 2024. OrcaLoca: Enhancing LLM Agents for Code Fault Localization. In Proceedings of the 46th International Conference on Software Engineering (ICSE).

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998–6008. DOI: 10.48550/arXiv.1706.03762

Yenisei D. Verdecia, Thelma E. Colanzi, Silvia R. Vergilio, and Marcelo C.B. dos Santos. 2017. An Enhanced Evaluation Model for Search-based Product Line Architecture Design.. In CIbSE. 155–168.

Chen Yang, Junjie Chen, Bin Lin, Jianyi Zhou, and Ziqi Wang. 2024. Enhancing LLM-based Test Generation for Hard-to-Cover Branches via Program Analysis. arXiv preprint arXiv:2404.04966 (2024). DOI: 10.48550/arXiv.2404.04966

Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocoderover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592–1604.