Metrics for Quality Assessment in Blockchain-based Systems: A Systematic Mapping Study

• Raphael Saraiva UECE
• Allysson Allex Araújo UFCA
• Pamella Soares UECE
• João Carlos Pontes UECE
• Jerffeson Souza UECE

DOI: https://doi.org/10.5753/sbsi.2025.246568

Resumo

Context: Blockchain technology has been widely adopted across various industries, such as finance, healthcare, and logistics, due to its ability to ensure transaction integrity and transparency. Problem: The growing adoption of blockchain systems demands rigorous quality assessment to ensure compliance with requirements. While software metrics for centralized systems are well-studied, a clear research gap remains in understanding their systematic adoption for evaluating blockchain-based systems. Solution: This study identifies, analyzes, and categorizes software metrics used in blockchain-based systems assessment grounded on the quality characteristics of the ISO/IEC 25010 standard. IS Theory: The study uses the ISO/IEC 25010 as a theoretical foundation for classifying and analyzing quality metrics. Method: A Systematic Mapping Study was conducted, identifying 128 primary studies extracted from different academic databases. Results: The findings highlight performance and security as fundamental to quality in blockchain systems, focusing on throughput and latency metrics. Additionally, Hyperledger Caliper is a frequent tool for evaluating blockchain network performance. Smart contracts, on the other hand, emerge as a promising area, given the lack of studies exploring their security and efficiency. Contributions: This work enhances the understanding of quality assessment in blockchain systems by mapping software metrics to the ISO/IEC 25010, covering predominant metrics, quality characteristics, and evaluation methods.

Palavras-chave: Blockchain, Quality Metrics, ISO/IEC 25010, Systematic Mapping Study

Referências

Rafa E Al-Qutaish. 2010. Quality models in software engineering literature: an analytical and comparative study. Journal of American Science 6, 3 (2010), 166–175.

Maher Alharby. 2023. Transaction latency within permissionless blockchains: analysis, improvement, and security considerations. Journal of Network and Systems Management 31, 1 (2023), 22.

Apostolos Ampatzoglou, Stamatia Bibi, Paris Avgeriou, Marijn Verbeek, and Alexander Chatzigeorgiou. 2019. Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Information and Software Technology 106 (2019), 201–230.

Andreas M Antonopoulos and Gavin Wood. 2004. Mastering Ethereum: Building Smart Contracts and Dapps. O’Reilly Media, Inc., USA. 415 pages.

Rafael Belchior, André Vasconcelos, Sérgio Guerreiro, and Miguel Correia. 2021. A survey on blockchain interoperability: Past, present, and future trends. Acm Computing Surveys (CSUR) 54, 8 (2021), 1–41.

Clodis Boscarioli, Renata Mendes de Araujo, Rita Suzana Maciel, Valdemar Vicente Graciano Neto, Flavio Oquendo, Elisa Yumi Nakagawa, Flavia Cristina Berrnardini, José Viterbo, Dalessandro Vianna, Carlos Bazilio Martins, et al. 2017. I GranDSI-BR: Grand Research Challenges in Information Systems in Brazil 2016-2026. (2017).

Vitalik Buterin et al. 2013. Ethereum white paper. GitHub repository 1 (2013), 22–23.

Sara Cohen and Aviv Zohar. 2018. Database perspectives on blockchains. arXiv preprint arXiv:1803.06015 (2018).

Marco Conoscenti, Antonio Vetro, and Juan Carlos De Martin. 2016. Blockchain for the Internet of Things: A systematic literature review. In 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA). IEEE, 1–6.

Emanuel F Coutinho, Wagner L Braga Bezerra, and Delano Maia. 2021. Uma Análise Inicial sobre a Aplicaçao de Blockchain na Sociedade. In Anais do II Workshop sobre as Implicações da Computação na Sociedade. SBC, 45–56.

Kyle Croman, Christian Decker, Ittay Eyal, Adem Efe Gencer, Ari Juels, Ahmed Kosba, Andrew Miller, Prateek Saxena, Elaine Shi, Emin Gün Sirer, et al. 2016. On Scaling Decentralized Blockchains: (A Position Paper). In International conference on financial cryptography and data security. Springer, 106–125.

Florian Deissenboeck, Elmar Juergens, Klaus Lochmann, and Stefan Wagner. 2009. Software quality models: Purposes, usage scenarios and requirements. In 2009 ICSE workshop on software quality. IEEE, 9–14.

John Estdale and Elli Georgiadou. 2018. Applying the ISO/IEC 25010 quality models to software product. In Systems, Software and Services Process Improvement: 25th European Conference, EuroSPI 2018, Bilbao, Spain, September 5-7, 2018, Proceedings 25. Springer, 492–503.

Markus Fynn, Alysson Bessani, and Fernando Pedone. 2020. Towards Blockchain Interoperability. Proceedings of the IEEE Symposium on Reliable Distributed Systems (2020). [link]

Xianyi Gao, Gradeigh D Clark, and Janne Lindqvist. 2016. Of two minds, multiple addresses, and one ledger: characterizing opinions, knowledge, and perceptions of Bitcoin across users and non-users. In Proceedings of the 2016 CHI conference on human factors in computing systems. 1656–1668.

ISO. 2023. ISO 25010:2023 - Blockchain and distributed ledger technologies — Reference architecture. [link] Accessed: 2024-11-12.

Simran Jaitly, Anil Kumar Mishra, and Latika Singh. 2014. A systematic review on the impact of metrics in software process improvement. Compusoft 3, 3 (2014), 624.

Mohamad Kassab, Joanna DeFranco, Tarek Malas, Valdemar Vicente Graciano Neto, and Giuseppe Destefanis. 2019. Blockchain: A panacea for electronic health records?. In 2019 IEEE/ACM 1st International Workshop on Software Engineering for Healthcare (SEH). IEEE, 21–24.

Staffs Keele et al. 2007. Guidelines for performing systematic literature reviews in software engineering. Technical Report. Technical report, ver. 2.3 ebse technical report. ebse.

Jiewu Leng, Man Zhou, J Leon Zhao, Yongfeng Huang, and Yiyang Bian. 2020. Blockchain security: A survey of techniques and research directions. IEEE Transactions on Services Computing 15, 4 (2020), 2490–2510.

Iuon-Chang Lin and Tzu-Chun Liao. 2017. A survey of blockchain security issues and challenges. Int. J. Netw. Secur. 19, 5 (2017), 653–659.

Matthew B Miles. 1994. Qualitative data analysis: An expanded sourcebook. Thousand Oaks (1994).

Arthur-Jozsef Molnar, Alexandra Neamţu, and Simona Motogna. 2020. Evaluation of software product quality metrics. In Evaluation of Novel Approaches to Software Engineering: 14th International Conference, ENASE 2019, Heraklion, Crete, Greece, May 4–5, 2019, Revised Selected Papers 14. Springer, 163–187.

Amrita Raj Mukker, Latika Singh, and Anil Kumar Mishra. 2014. Systematic review of metrics in software agile projects. Compusoft 3, 2 (2014), 533–539.

Międzynarodowa Organizacja Normalizacyjna. 2011. Systems and Software Engineering-Systems and Software Quality Requirements and Evaluation (SQuaRE)- System and Software Quality Models. ISO.

James A O’Brien and GM Marakas. 2011. Developing business/IT solutions. Management information systems 488489 (2011), 74–89.

Marco Ortu, Matteo Orrú, and Giuseppe Destefanis. 2019. On comparing software quality metrics of traditional vs blockchain-oriented software: An empirical study. In 2019 IEEE International Workshop on Blockchain Oriented Software Engineering (IWBOSE). IEEE, 32–37.

Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael Mattsson. 2008. Systematic mapping studies in software engineering. In 12th international conference on evaluation and assessment in software engineering (EASE). BCS Learning & Development.

Giuseppe Antonio Pierro, Roberto Tonelli, and Michele Marchesi. 2020. Smart-corpus: an organized repository of ethereum smart contracts source code and metrics. arXiv preprint arXiv:2011.01723 (2020).

Simone Porru, Andrea Pinna, Michele Marchesi, and Roberto Tonelli. 2017. Blockchain-oriented software engineering: challenges and new directions. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C). IEEE, 169–171.

Benedikt Putz and Günther Pernul. 2020. Detecting blockchain security threats. In 2020 IEEE International Conference on Blockchain (Blockchain). IEEE, 313–320.

Attia Qammar, Ahmad Karim, Huansheng Ning, and Jianguo Ding. 2023. Securing federated learning with blockchain: a systematic literature review. Artificial Intelligence Review 56, 5 (2023), 3951–3985.

Siraj Raval. 2016. Decentralized applications: harnessing Bitcoin’s blockchain technology. " O’Reilly Media, Inc.".

R. Saraiva, A. A. Araújo, P. Soares, Pontes J. C., and J. Souza. 2024. Supplementary material of Metrics for Quality Assessment in Blockchain-based Systems: A Systematic Mapping Study. [link].

Rajesh Soundararajan and VM Shenbagaraman. 2023. Unlocking the Potential of Blockchain Through Multi-Criteria Decision Making in Platform Selection. International Journal of Professional Business Review: Int. J. Prof. Bus. Rev. 8, 4 (2023), 49.

Don Tapscott and Alex Tapscott. 2016. Blockchain revolution: how the technology behind bitcoin is changing money, business, and the world. Penguin.

Roberto Tonelli, Giuseppe Destefanis, Michele Marchesi, and Marco Ortu. 2018. Smart contracts software metrics: a first study. arXiv preprint arXiv:1802.01517 (2018).

Gavin Wood et al. 2014. Ethereum: A secure decentralised generalised transaction ledger. Ethereum project yellow paper 151, 2014 (2014), 1–32.

Xiwei Xu, Ingo Weber, and Mark Staples. 2019. Architecture for blockchain applications. Springer.

Xiwei Xu, Ingo Weber, Mark Staples, Xiwei Xu, Ingo Weber, and Mark Staples. 2019. Blockchain patterns. Architecture for Blockchain Applications (2019), 113–148.

Jesse Yli-Huumo, Deokyoon Ko, Sujin Choi, Sooyong Park, and Kari Smolander. 2016. Where is current research on blockchain technology?—a systematic review. PloS one 11, 10 (2016), e0163477.

Zibin Zheng, Shaoan Xie, Hongning Dai, Xiangping Chen, and Huaimin Wang. 2017. An overview of blockchain technology: Architecture, consensus, and future trends. In 2017 IEEE international congress on big data (BigData congress). Ieee, 557–564.

Sisi Zhou, Kuanching Li, Lijun Xiao, Jiahong Cai, Wei Liang, and Arcangelo Castiglione. 2023. A systematic review of consensus mechanisms in blockchain. Mathematics 11, 10 (2023), 2248.

Mengmeng Zhu and Hoang Pham. 2022. Software reliability modeling and methods: A state of the art review. Optimization Models in Software Reliability (2022), 1–29.