Benchmarking Agentic Frameworks for Incremental Software Evolution
Resumo
Agentic frameworks are increasingly used in software engineering, but comparing them remains difficult due to differences in goals, output formats, and evaluation practices. This paper extends the evaluation of Business Autonomous Entities (BAEs), an approach for incremental software evolution, through a controlled benchmark against five agentic software engineering frameworks: ChatDev, GitHub Spec Kit, MetaGPT, AutoGen, and Self-Organized Agents (SOA). All frameworks evolve the same Student–Course system through six incremental changes, with 50 executions per framework and 300 runs in total. We analyze execution time, token usage, monetary cost, API calls, cached tokens, and sounds good, a binary structural sanity metric for checking whether generated artifacts satisfy a minimal set of prompt requirements. Results show that no framework dominates all dimensions. ChatDev obtains the highest sounds good rate, but at higher cost and execution time. BAEs and AutoGen achieve nearly the same sounds good rate while remaining faster and cheaper, and BAEs show a more stable cost profile. These findings suggest that agentic software engineering frameworks should be evaluated not only by raw efficiency, but also by artifact usability, stability, and support for incremental evolution.
Referências
Frederick P. Brooks. 1987. No Silver Bullet Essence and Accidents of Software Engineering. Computer 20, 4 (1987), 10–19. DOI: 10.1109/MC.1987.1663532
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Hodas, Sandhini Chen, Alec Radford, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021).
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE, 31–53.
GitHub. 2026. Spec Kit: A toolkit for Spec-Driven Development. [link]
Anderson Martins Gomes, Paulo Henrique M. Maia, Vitor Fontenele de Oliveira Linhares, and Ingrid Lima Vieira. 2026. From Stateless Code Generation to Living Ontologies: Business Autonomous Entities in Action. In Proceedings of the 2026 International Workshop on Agentic Engineering (AGENT ’26). Association for Computing Machinery, New York, NY, USA, 125–134. DOI: 10.1145/3786167.3788408
JundaHe, Christoph Treude, andDavid Lo. 2025. LLM-BasedMulti-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead. ACM Transactions on Software Engineering and Methodology 34, 5 (2025), 1–30. DOI: 10.1145/3712003
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. 2023. MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations.
Yoichi Ishibashi and Yoshimasa Nishimura. 2024. Self-organized agents: A llm multi-agent framework toward ultra large-scale code generation and optimization. arXiv preprint arXiv:2404.02183 (2024).
M. M. Lehman. 1996. Laws of Software Evolution Revisited. In Proceedings of the 5th European Workshop on Software Process Technology (EWSPT ’96). Springer-Verlag, Berlin, Heidelberg, 108–124.
Ingrid Lima, Vitor Linhares, Anderson Martins Gomes, and Paulo Henrique Maia. 2026. A Catalogue of Evaluation Metrics for LLM-BasedMulti-Agent Frameworks in Software Engineering. In Proceedings of the 2026 International Workshop on Agentic Engineering (AGENT ’26). Association for Computing Machinery, New York, NY, USA, 120–124. DOI: 10.1145/3786167.3788430
Mahmoud Mohammadi, Yipeng Li, Jane Lo, and Wendy Yip. 2025. Evaluation and Benchmarking of LLM Agents: A Survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (Toronto ON, Canada) (KDD ’25). Association for Computing Machinery, New York, NY, USA, 6129–6139. DOI: 10.1145/3711896.3736570
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. 2023. Chatdev: Communicative agents for software development. arXiv preprint arXiv:2307.07924 (2023).
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. In International Conference on Learning Representations, Vol. 2024. 25055–25083.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. 2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling.
