Building AI-Intensive Software with AI: Early Results and a Cautionary Tale on Measuring Development Cost
Resumo
Empirical reports on the true cost of AI-intensive software development remain scarce, and the few that exist are easy to get wrong in ways that never surface in the final number. We report early results from an ongoing case study: a six-person student team built a full conversational onboarding assistant — RAG-based code chat, guided tours, dependency graphs, technical-debt analysis — over one academic term using pervasive AI assistance. We instrumented development with a three-layer cost model (real AI spend, self-reported human effort, human counterfactual) and initially reported a 19.4× cost ratio. A follow-up pass revealed two independent errors — inferring per-token cost under a flat-rate subscription, and pricing the counterfactual with the wrong regional labor rates — that together had inflated the ratio by roughly 2×; the corrected figure is ∼9.9×.We present this correction as an early, generalizable finding in its own right: both errors are easy to make, invisible in the final number, and plausibly common in similar reports. We outline next steps toward a more robust, replicable costing methodology for AI-intensive development.
Referências
Jim Buchan, Stephen G. MacDonell, and Jennifer Yang. 2019. Effective team onboarding in Agile software development: techniques and goals. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1–11. DOI: 10.1109/ESEM.2019.8870189
Fabian Fagerholm and Jurgen Munch. 2012. Developer experience: Concept and definition. In 2012 International Conference on Software and System Process (ICSSP). IEEE, 73–77. DOI: 10.1109/ICSSP.2012.6225984
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). DOI: 10.1109/ICSE-FoSE59343.2023.00008
Nuruzzaman Faruqui, Priyabrata Thatoi, Rohit Choudhary, Ivana Roncevic, Hamed Alqahtani, Iqbal H Sarker, and Shapla Khanam. 2024. AI-analyst: An AI-assisted SDLC analysis framework for business cost optimization. IEEE Access 12 (2024), 195188–195203.
Peggy Gregory, Diane E. Strode, Helen Sharp, and Leonor Barroca. 2022. An onboarding model for integrating newcomers into agile project teams. Information and Software Technology 143 (mar 2022), 106792. DOI: 10.1016/j.infsof.2021.106792
Michaela Greiler, Margaret-Anne Storey, and Abi Noda. 2023. An Actionable Framework for Understanding and Improving Developer Experience. IEEE Transactions on Software Engineering 49, 4 (apr 2023), 1411–1425. DOI: 10.1109/TSE.2022.3175660
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Transactions on Software Engineering and Methodology 33, 8 (2024). DOI: 10.1145/3695988
An Ju, Hitesh Sajnani, Scot Kelly, and Kim Herzig. 2021. A Case Study of Onboarding in Software Teams: Tasks and Strategies. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 613–623. DOI: 10.1109/ICSE43902.2021.00063
Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an LLM to Help with Code Understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE). DOI: 10.1145/3597503.3639187
Elise Paradis, Kate Grey, Quinn Madison, Daye Nam, Andrew Macvean, Vahid Meimand, Nan Zhang, Ben Ferrari-Church, and Satish Chandra. 2025. How much does AI impact development speed? An enterprise-based randomized controlled trial. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 618–629.
Jaakko Sauvola, Sasu Tarkoma, Mika Klemettinen, Jukka Riekki, and David Doermann. 2024. Future of Software Development with Generative AI. Automated Software Engineering 31 (2024). DOI: 10.1007/s10515-024-00426-z
