Symbolic AI versus LLMs for Cross-Format COSMIC Functional Size Measurement: A Five-Method Benchmark

• Nathanael Motta UPE
• Cleyton Mário de Oliveira Rodrigues UPE
• Carlo Marcelo Revoredo da Silva UPE

DOI: https://doi.org/10.5753/eniac.2026.30923

Resumo

Automated COSMIC functional size measurement reaches near-expert accuracy with Large Language Models (LLMs), yet each method is validated on a single requirement format, so heterogeneity is untested. Our five-method, four-format heterogeneity matrix pits language models against a zero-cost, auditable symbolic pipeline. Specializing only a per-format normalization layer over an invariant engine, the framework has no significantly negative Standardized Accuracy cell (significantly positive in three of four), whereas every frozen competitor collapses off-format; a small adapter beats GPT-4-turbo on user stories without touching the engine. Heterogeneity is resolved by normalization rather than the engine, and the same layer rescues the LLM.

Referências

Abran, A. (2015). Software Project Estimation: The Fundamentals for Providing High-Quality Information to Decision Makers. John Wiley & Sons, Hoboken, NJ, USA.

Albrecht, A. J. and Gaffney, J. E. (1983). Software function, source lines of code, and development effort prediction: A software science validation. IEEE Transactions on Software Engineering, SE-9(6):639–648.

Bagriyanik, S. and Karahoca, A. (2016). Automated COSMIC function point measurement using a requirements engineering ontology. Information and Software Technology, 72:189–203.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46.

COSMIC (2009). The COSMIC Functional Size Measurement Method, Version 3.0.1: Measurement Manual. The Common Software Measurement International Consortium.

COSMIC Measurement Practices Committee (2019). COSMIC case studies: ACME car hire, RestoSys, and course registration (C-REG). cosmic-sizing.org.

Dalpiaz, F. (2018). Requirements data sets (user stories). V1. DOI: 10.17632/7zbk8zsd8y.1.

De Vito, G., Di Martino, S., Ferrucci, F., Gravino, C., and Palomba, F. (2025). LLM-based automation of COSMIC functional size measurement from use cases. IEEE Transactions on Software Engineering, 51(5):1500–1523.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382.

Honnibal, M. and Montani, I. (2017). spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. Software: [link].

Hussain, I., Kosseim, L., and Ormandjieva, O. (2013). Approximation of COSMIC functional size to support early effort estimation in Agile. Data & Knowledge Engineering, 85:2–14.

ISO/IEC (2011). ISO/IEC 19761:2011 — software engineering — COSMIC: a functional size measurement method.

Jurkiewicz, J. and Nawrocki, J. (2015). Automated events identification in use cases. Information and Software Technology, 58:110–122.

Laqrichi, S. (2024). A hybrid framework for COSMIC measurement: Combining large language models with a rule-based system. In Joint Proceedings of IWSM-Mensura 2024, volume 3852 of CEUR Workshop Proceedings, Montréal, Canada.

Ochodek, M., Kopczyńska, S., and Staron, M. (2020). Deep learning model for end-to-end approximation of COSMIC functional size based on use-case names. Information and Software Technology, 123:106310.

Qwen Team (2025). Qwen3 technical report. arXiv:2505.09388.

Radliński, Ł. and Swacha, J. (2025). Large language models for early-stage software project estimation: A systematic mapping study. Applied Sciences, 15(24):13099.

Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). “why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144. ACM.

Shepperd, M. and MacDonell, S. (2012). Evaluating prediction systems in software project estimation. Information and Software Technology, 54(8):820–827.

Ungan, E., Hammond, C. A., and Abran, A. (2018). Automated COSMIC measurement and requirement quality improvement through ScopeMaster tool. In Proceedings of IWSM-Mensura 2018, volume 2207 of CEUR Workshop Proceedings.

Ünlü, H., Tenekeci, S., Çiftçi, C., et al. (2024). Predicting software functional size using natural language processing: An exploratory case study. In 2024 50th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pages 188–193. IEEE.

Yokogawa, K., Hiroishi, M., Amasaki, S., Aman, H., and Yokogawa, T. (2024). A comparative study on COSMIC FP approximation with deep learning and conventional machine learning. In 2024 IEEE/ACIS 22nd International Conference on Software Engineering Research, Management and Applications (SERA), pages 346–351. IEEE.

Zhang, C., Tong, S., Mo, W., Zhou, Y., Xia, Y., and Shen, B. (2016). ESSE: An early software size estimation method based on auto-extracted requirements features. In Proceedings of the 8th Asia-Pacific Symposium on Internetware, pages 112–115. ACM.

Zhao, Z., Jiang, H., Zhao, R., and He, B. (2024). Emergence of a novel domain expert: A generative AI-based framework for software function point analysis. In 2024 39th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 2245–2250. IEEE.