Agreement-Based Confidence for LLM-Based Functional Classification of User Stories
Resumo
Classifying user stories into functional categories can support backlog organization, reuse, and requirements analysis, but manual classification is costly and difficult to scale. Large Language Models (LLMs) can automate part of this task, although accuracy alone is insufficient for practical adoption: analysts also need to know when a classification should be accepted, reviewed, or corrected. This paper presents a controlled empirical study of confidence-aware user-story classification with GPT-5.2, DeepSeek, and Llama. The models classified 191 user stories from 10 projects using the same prompt, taxonomy, output schema, and execution constraints, with three repeated runs per model. The study evaluates exact classification performance, repeated-run stability, and inter-model agreement as a practical confidence signal. Results show that model accuracy and stability do not necessarily coincide. GPT-5.2 achieved the highest exact accuracy, while DeepSeek was the most stable across runs. Agreement among models provided useful evidence for selective automation: when all three models agreed, exact accuracy reached 84.4%; when only two agreed, it dropped to 47.8%; and when all models disagreed, it dropped to 22.2%. The results also show that disagreement is associated with underspecified stories, multi-intent stories, out-of-coverage cases, and problematic taxonomy labels. These findings suggest that multiple LLMs can be used as complementary decision-support agents rather than as isolated classifiers.
Referências
Anis R. Amna and Geert Poels. 2022. Ambiguity in User Stories: A Systematic Literature Review. Information and Software Technology 145 (2022), 106824. DOI: 10.1016/j.infsof.2022.106824
Anis Rahmawati Amna and Geert Poels. 2022. Systematic Literature Mapping of User Story Research. IEEE Access 10 (2022), 51723–51746. DOI: 10.1109/ACCESS.2022.3173745
Edna Canedo, Angelica Calazans, Geovana Silva, Eloisa Masson, and Isabel Brito. 2024. On the Challenges to Documenting Requirements in Agile Software Development: A Practitioners’ Perspective. In Anais do XXVII Congresso Ibero-Americano em Engenharia de Software. SBC, 286–300. DOI: 10.5753/cibse.2024.28454
Porchourng Chuor, Ashwin Ittoo, and Samedi Heng. 2024. User Story Classification with Machine Learning and LLMs. In Knowledge Science, Engineering and Management: 17th International Conference, KSEM 2024, Birmingham, UK, August 16–18, 2024, Proceedings, Part I (Birmingham, United Kingdom). Springer-Verlag, 161–175. DOI: 10.1007/978-981-97-5492-2_13
Ednaldo Dilorenzo, Emanuel Dantas, Mirko Perkusich, Felipe Ramos, Alexandre Costa, Danyllo Albuquerque, Hyggo Almeida, and Angelo Perkusich. 2020. Enabling the Reuse of Software Development Assets Through a Taxonomy for User Stories. IEEE Access 8 (2020), 107285–107300. DOI: 10.1109/ACCESS.2020.2996951
Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 4885–4894.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70). 1321–1330.
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv:2308.10620 [cs.SE] [link]
Garm Lucassen, Fabiano Dalpiaz, Jan Martijn E. M. van der Werf, and Sjaak Brinkkemper. 2016. Improving Agile Requirements: The Quality User Story Framework and Tool. Requirements Engineering 21, 3 (2016), 383–403. DOI: 10.1007/s00766-016-0250-x
Johannes J. Norheim, Eric Rebentisch, Dekai Xiao, Lorenz Draeger, Alain Kerbrat, and Olivier L. de Weck. 2024. Challenges in applying large language models to requirements engineering tasks. Design Science 10 (2024), e16. DOI: 10.1017/dsj.2024.8
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16). Association for Computing Machinery, New York, NY, USA, 1135–1144. DOI: 10.1145/2939672.2939778
Tor Sporsem, Torgeir Dingsøyr, and Klaas-Jan Stol. 2026. User stories as boundary objects in agile requirements engineering: A theoretical literature review. Journal of Systems and Software 233 (2026), 112693. DOI: 10.1016/j.jss.2025.112693
Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in Software Engineering. Springer. DOI: 10.1007/978-3-642-29044-2
Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Shengcheng Yu, Weisong Sun, Yun Yang, and Zhenyu Chen. 2026. A Survey on Large Language Models for Software Engineering. Science China Information Sciences 69, 4 (March 2026), 141102. DOI: 10.1007/s11432-025-4670-0
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, Vol. 36. 46595–46623.
