Metrics in Large-Scale Agile Software Development: A Multivocal Literature Review
Resumo
Software development is widespread across various sectors. As large-scale projects increasingly adopt agile development practices, there arises a need for metrics to enhance team coordination, promote continuous improvement and monitor progress. This discussion focuses on the current state of metrics for large-scale agile software development, outlining the reasons for their adoption and showcasing the achieved results. The analysis involves a comprehensive literature review, exploring grey literature. A catalog of metrics applicable to scalable agile projects is presented, featuring examples such as ‘Velocity’, ‘Business value per effort’, and ‘Defect rate’.
Referências
Basili, V. R. (1992). Software modeling and measurement: the goal/question/metric paradigm. Technical report.
Britto, R., Usman, M., and Mendes, E. (2014). Effort estimation in agile global software development context. In Agile Methods. Large-Scale Development, Refactoring, Testing, and Estimation: XP 2014 International Workshops, Rome, Italy, May 26-30, 2014, Revised Selected Papers 15, pages 182–192. Springer.
Brown, A. W. (2011). A case study in agile-at-scale delivery. In International Conference on Agile Software Development, pages 266–281. Springer.
Camara, R., Alves, A., Monte, I., and Marinho, M. (2020). Agile global software development: A systematic literature review. In Proceedings of the XXXIV Brazilian Symposium on Software Engineering, pages 31–40.
Camara, R., Marinho, M., and Moura, H. (2021). Agile tailoring with scaling agile frameworks in distributed large-scale settings. In Anais Estendidos do XX Simpósio Brasileiro de Qualidade de Software, pages 12–17. SBC.
Chloros, D., Gerogiannis, V. C., and Kakarontzas, G. (2022). Use of software and project management metrics in agile software development methodologies: A systematic mapping study. In Proceedings of the 2022 European Symposium on Software Engineering, pages 25–32.
Copado, T. (2022). How to optimize flow metrics quality in scaled agile framework (safe). [link]. (accessed 4 January 2023).
Dingsøyr, T., Fægri, T. E., and Itkonen, J. (2014). What is large in large-scale? a taxonomy of scale for agile software development. In Product-Focused Software Process Improvement: 15th International Conference, PROFES 2014, Helsinki, Finland, December 10-12, 2014. Proceedings 15, pages 273–276. Springer.
Dingsøyr, T. and Moe, N. B. (2014). Towards principles of large-scale agile development. In International Conference on Agile Software Development, pages 1–8. Springer.
Edison, H., Wang, X., and Conboy, K. (2021). Comparing methods for large-scale agile software development: A systematic literature review. IEEE Transactions on Software Engineering.
Ertaban, C., Sarikaya, E., and Bagriyanik, S. (2018). Agile performance indicators for team performance evaluation in a corporate environment. In Proceedings of the 19th International Conference on Agile Software Development: Companion, pages 1–3.
Fenton, N. and Bieman, J. (2014). Software metrics: a rigorous and practical approach. CRC press.
Garousi, V., Felderer, M., and Mäntylä, M. V. (2019). Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and software technology, 106:101–121.
Greening, D. R. (2010). Enterprise scrum: Scaling scrum to the executive level. In 2010 43rd Hawaii International Conference on System Sciences, pages 1–10. IEEE.
Greening, D. R. (2015). Agile enterprise metrics. In 2015 48th Hawaii International Conference on System Sciences, pages 5038–5044. IEEE.
Grimaldi, P., Perrotta, L., Corvello, V., and Verteramo, S. (2016). An agile, measurable and scalable approach to deliver software applications in a large enterprise. International Journal of Agile Systems and Management, 9(4):326–339.
Heidenberg, J., Weijola, M., Mikkonen, K., and Porres, I. (2013). A metrics model to measure the impact of an agile transformation in large software development organizations. In Int. Conference on Agile Software Development, pages 165–179. Springer.
Hossain, S. S., Ahmed, P., and Arafat, Y. (2021). Software process metrics in agile software development: A systematic mapping study. In Computational Science and Its Applications–ICCSA 2021: 21st International Conference, Cagliari, Italy, September 13–16, 2021, Proceedings, Part IX 21, pages 15–26. Springer.
Kišš, F. and Rossi, B. (2018). Agile to lean software development transformation: A systematic literature review. In 2018 Federated Conference on Computer Science and Information Systems (FedCSIS), pages 969–973. IEEE.
Korpivaara, I., Tuunanen, T., and Seppänen, V. (2021). Performance measurement in scaled agile organizations. In Proceedings of the Annual Hawaii International Conference on System Sciences. University of Hawai’i at Manoa.
Korson, T. (2015). Cash-in comfort. [link]. (accessed 4 January 2023].
Laanti, M. (2014). Characteristics and principles of scaled agile. In International Conference on Agile Software Development, pages 9–20. Springer.
Leal, S., Hauck, J., Bertan, M., and Vieira, G. (2022). How agile organizations use metrics: A systematic literature mapping. In Proceedings of the XXI Brazilian Symposium on Software Quality, pages 1–11.
Liyanage, J. R. (2014). Lean agile metrics for scaled agile systems. [link]. (accessed 4 January 2023).
Marinho, M., Camara, R., and Sampaio, S. (2021). Toward unveiling how safe framework supports agile in global software development. IEEE Access, 9:109671–109692.
Marinho, M., Noll, J., Richardson, I., and Beecham, S. (2019). Plan-driven approaches are alive and kicking in agile global software development. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 1–11. IEEE.
Menezes, R., Marinho, M., and Sampaio, S. (2024a). Complete metrics tables for “goal question metrics” modeling. DOI: 10.7910/DVN/HTQTEE.
Menezes, R., Marinho, M., and Sampaio, S. (2024b). Replication data for: Metrics in large-scale agile software development: A multivocal literature review. 10.7910/DVN/8MJG8K.
Petersen, K., Feldt, R., Mujtaba, S., and Mattsson, M. (2008). Systematic mapping studies in software engineering. In 12th International Conference on Evaluation and Assessment in Software Engineering (EASE) 12, pages 1–10.
Philipp, P., Tobisch, F., Menzel, L., and Matthes, F. (2023). A method for metric management at a large-scale agile software development organization.
Razzak, M. A., Noll, J., Richardson, I., Canna, C. N., and Beecham, S. (2017). Transition from plan driven to safe®: periodic team self-assessment. In International Conference on Product-Focused Software Process Improvement, pages 573–585. Springer.
SAFe (2022). Measure and grow. [link]. (accessed 4 January 2023).
Scrum@Scale (2022). The scrum at scale® guide. the definitive guide to the scrum@scale framework. [link]. (accessed 4 January 2023).
Shameem, M., Kumar, C., Chandra, B., and Khan, A. A. (2017). Systematic review of success factors for scaling agile methods in global software development environment: A client-vendor perspective. In 2017 24th Asia-Pacific Software Engineering Conference Workshops (APSECW), pages 17–24. IEEE.
Staron, M., Meding, W., and Caiman, M. (2013). Improving completeness of measurement systems for monitoring software development workflows. In International Conference on Software Quality, pages 230–243. Springer.
Staron, M., Meding, W., and Palm, K. (2012). Release readiness indicator for mature agile and lean software development projects. In International Conference on Agile Software Development, pages 93–107. Springer.
Stettina, C. J. and Schoemaker, L. (2018). Reporting in agile portfolio management: Routines, metrics and artefacts to maintain an effective oversight. In International Conference on Agile Software Development, pages 199–215. Springer, Cham.
Tabib, R. (2013). Need 4 speed: leverage new metrics to boost your velocity without compromising on quality. In 2013 Agile Conference, pages 117–120. IEEE.
Tessarolo, F., Petsani, D., Conotter, V., Nollo, G., Conti, G., Nikolaidou, M., Onorati, G., Bamidis, P. D., and Konstantinidis, E. I. (2022). Developing ambient assisted living technologies exploiting potential of user-centred co-creation and agile methodology: the captain project experience. Journal of Ambient Intelligence and Humanized Computing, pages 1–16.
Thawaba, A. A., Ramli, A. A., Fudzee, M., Md, F., and Wadata, J. (2020). A mechanism to support agile frameworks enhancing reliability assessment for scs development: a case study of medical surgery departments. In International Conference on Soft Computing and Data Mining, pages 66–76. Springer.
Tripathi, N., Rodríguez, P., Ahmad, M. O., and Oivo, M. (2015). Scaling kanban for software development in a multisite organization: challenges and potential solutions. In International Conference on Agile Software Development, pages 178–190. Springer.
Wieringa, R., Maiden, N., Mead, N., and Rolland, C. (2006). Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requirements engineering, 11:102–107.
Britto, R., Usman, M., and Mendes, E. (2014). Effort estimation in agile global software development context. In Agile Methods. Large-Scale Development, Refactoring, Testing, and Estimation: XP 2014 International Workshops, Rome, Italy, May 26-30, 2014, Revised Selected Papers 15, pages 182–192. Springer.
Brown, A. W. (2011). A case study in agile-at-scale delivery. In International Conference on Agile Software Development, pages 266–281. Springer.
Camara, R., Alves, A., Monte, I., and Marinho, M. (2020). Agile global software development: A systematic literature review. In Proceedings of the XXXIV Brazilian Symposium on Software Engineering, pages 31–40.
Camara, R., Marinho, M., and Moura, H. (2021). Agile tailoring with scaling agile frameworks in distributed large-scale settings. In Anais Estendidos do XX Simpósio Brasileiro de Qualidade de Software, pages 12–17. SBC.
Chloros, D., Gerogiannis, V. C., and Kakarontzas, G. (2022). Use of software and project management metrics in agile software development methodologies: A systematic mapping study. In Proceedings of the 2022 European Symposium on Software Engineering, pages 25–32.
Copado, T. (2022). How to optimize flow metrics quality in scaled agile framework (safe). [link]. (accessed 4 January 2023).
Dingsøyr, T., Fægri, T. E., and Itkonen, J. (2014). What is large in large-scale? a taxonomy of scale for agile software development. In Product-Focused Software Process Improvement: 15th International Conference, PROFES 2014, Helsinki, Finland, December 10-12, 2014. Proceedings 15, pages 273–276. Springer.
Dingsøyr, T. and Moe, N. B. (2014). Towards principles of large-scale agile development. In International Conference on Agile Software Development, pages 1–8. Springer.
Edison, H., Wang, X., and Conboy, K. (2021). Comparing methods for large-scale agile software development: A systematic literature review. IEEE Transactions on Software Engineering.
Ertaban, C., Sarikaya, E., and Bagriyanik, S. (2018). Agile performance indicators for team performance evaluation in a corporate environment. In Proceedings of the 19th International Conference on Agile Software Development: Companion, pages 1–3.
Fenton, N. and Bieman, J. (2014). Software metrics: a rigorous and practical approach. CRC press.
Garousi, V., Felderer, M., and Mäntylä, M. V. (2019). Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and software technology, 106:101–121.
Greening, D. R. (2010). Enterprise scrum: Scaling scrum to the executive level. In 2010 43rd Hawaii International Conference on System Sciences, pages 1–10. IEEE.
Greening, D. R. (2015). Agile enterprise metrics. In 2015 48th Hawaii International Conference on System Sciences, pages 5038–5044. IEEE.
Grimaldi, P., Perrotta, L., Corvello, V., and Verteramo, S. (2016). An agile, measurable and scalable approach to deliver software applications in a large enterprise. International Journal of Agile Systems and Management, 9(4):326–339.
Heidenberg, J., Weijola, M., Mikkonen, K., and Porres, I. (2013). A metrics model to measure the impact of an agile transformation in large software development organizations. In Int. Conference on Agile Software Development, pages 165–179. Springer.
Hossain, S. S., Ahmed, P., and Arafat, Y. (2021). Software process metrics in agile software development: A systematic mapping study. In Computational Science and Its Applications–ICCSA 2021: 21st International Conference, Cagliari, Italy, September 13–16, 2021, Proceedings, Part IX 21, pages 15–26. Springer.
Kišš, F. and Rossi, B. (2018). Agile to lean software development transformation: A systematic literature review. In 2018 Federated Conference on Computer Science and Information Systems (FedCSIS), pages 969–973. IEEE.
Korpivaara, I., Tuunanen, T., and Seppänen, V. (2021). Performance measurement in scaled agile organizations. In Proceedings of the Annual Hawaii International Conference on System Sciences. University of Hawai’i at Manoa.
Korson, T. (2015). Cash-in comfort. [link]. (accessed 4 January 2023].
Laanti, M. (2014). Characteristics and principles of scaled agile. In International Conference on Agile Software Development, pages 9–20. Springer.
Leal, S., Hauck, J., Bertan, M., and Vieira, G. (2022). How agile organizations use metrics: A systematic literature mapping. In Proceedings of the XXI Brazilian Symposium on Software Quality, pages 1–11.
Liyanage, J. R. (2014). Lean agile metrics for scaled agile systems. [link]. (accessed 4 January 2023).
Marinho, M., Camara, R., and Sampaio, S. (2021). Toward unveiling how safe framework supports agile in global software development. IEEE Access, 9:109671–109692.
Marinho, M., Noll, J., Richardson, I., and Beecham, S. (2019). Plan-driven approaches are alive and kicking in agile global software development. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 1–11. IEEE.
Menezes, R., Marinho, M., and Sampaio, S. (2024a). Complete metrics tables for “goal question metrics” modeling. DOI: 10.7910/DVN/HTQTEE.
Menezes, R., Marinho, M., and Sampaio, S. (2024b). Replication data for: Metrics in large-scale agile software development: A multivocal literature review. 10.7910/DVN/8MJG8K.
Petersen, K., Feldt, R., Mujtaba, S., and Mattsson, M. (2008). Systematic mapping studies in software engineering. In 12th International Conference on Evaluation and Assessment in Software Engineering (EASE) 12, pages 1–10.
Philipp, P., Tobisch, F., Menzel, L., and Matthes, F. (2023). A method for metric management at a large-scale agile software development organization.
Razzak, M. A., Noll, J., Richardson, I., Canna, C. N., and Beecham, S. (2017). Transition from plan driven to safe®: periodic team self-assessment. In International Conference on Product-Focused Software Process Improvement, pages 573–585. Springer.
SAFe (2022). Measure and grow. [link]. (accessed 4 January 2023).
Scrum@Scale (2022). The scrum at scale® guide. the definitive guide to the scrum@scale framework. [link]. (accessed 4 January 2023).
Shameem, M., Kumar, C., Chandra, B., and Khan, A. A. (2017). Systematic review of success factors for scaling agile methods in global software development environment: A client-vendor perspective. In 2017 24th Asia-Pacific Software Engineering Conference Workshops (APSECW), pages 17–24. IEEE.
Staron, M., Meding, W., and Caiman, M. (2013). Improving completeness of measurement systems for monitoring software development workflows. In International Conference on Software Quality, pages 230–243. Springer.
Staron, M., Meding, W., and Palm, K. (2012). Release readiness indicator for mature agile and lean software development projects. In International Conference on Agile Software Development, pages 93–107. Springer.
Stettina, C. J. and Schoemaker, L. (2018). Reporting in agile portfolio management: Routines, metrics and artefacts to maintain an effective oversight. In International Conference on Agile Software Development, pages 199–215. Springer, Cham.
Tabib, R. (2013). Need 4 speed: leverage new metrics to boost your velocity without compromising on quality. In 2013 Agile Conference, pages 117–120. IEEE.
Tessarolo, F., Petsani, D., Conotter, V., Nollo, G., Conti, G., Nikolaidou, M., Onorati, G., Bamidis, P. D., and Konstantinidis, E. I. (2022). Developing ambient assisted living technologies exploiting potential of user-centred co-creation and agile methodology: the captain project experience. Journal of Ambient Intelligence and Humanized Computing, pages 1–16.
Thawaba, A. A., Ramli, A. A., Fudzee, M., Md, F., and Wadata, J. (2020). A mechanism to support agile frameworks enhancing reliability assessment for scs development: a case study of medical surgery departments. In International Conference on Soft Computing and Data Mining, pages 66–76. Springer.
Tripathi, N., Rodríguez, P., Ahmad, M. O., and Oivo, M. (2015). Scaling kanban for software development in a multisite organization: challenges and potential solutions. In International Conference on Agile Software Development, pages 178–190. Springer.
Wieringa, R., Maiden, N., Mead, N., and Rolland, C. (2006). Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requirements engineering, 11:102–107.
Publicado
06/05/2024
Como Citar
MENEZES, Renato; MARINHO, Marcelo; SAMPAIO, Suzana.
Metrics in Large-Scale Agile Software Development: A Multivocal Literature Review. In: CONGRESSO IBERO-AMERICANO EM ENGENHARIA DE SOFTWARE (CIBSE), 27. , 2024, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 106-120.
DOI: https://doi.org/10.5753/cibse.2024.28442.
