Experimental Evaluation of FMCheck: A Replication Study
Resumo
Software Product Lines are usually specified using feature models. A hierarchically arranged set of features with different relationships among them represents a feature model. However, there is a lack of techniques to support the detection of semantic defects in feature models. In this context, it was recently developed FMCheck, a checklist-based inspection technique to support the detection of defects in feature models. The results of a first study conducted by FMCheck's developers indicated its feasibility (more effective) when compared to ad-hoc techniques. This paper reports the replication accomplished by an independent research group following a different experimental design but using the same artifacts. The obtained results strengthened the previous findings, indicating that FMCheck is more effective than ad-hoc inspections. However, additional replications should be performed with different experimental designs to understand better the influence of the artifacts inspected over such findings.
Palavras-chave:
Experimental, Evaluation, FMCheck
Referências
Batory, D., Benavides, D. and Ruiz-Cortés, A. (2006) “Automated analysis of feature models: challenges ahead,” Communication ACM, vol. 49, pp. 45-47.
Benavides, D., Segura, S., Trinidad, P. and Ruiz-Cortés, A. (2007) “Fama: Tooling a framework for the automated analysis of feature models,” First International Workshop on Variability Modeling of Software-Intensive Systems, pp. 129-134.
Benavides, D., Segura, S. and Ruiz-Cortés, A. (2010) “Automated analysis of feature models 20 years later: A literature review,” Info System, vol. 35, no. 6, pp. 615-636.
Blois, A. P. T. B., de Oliveira, R. F., Maia, N., Werner, C. and Becker, K. (2006) “Variability modeling in a component-based domain engineering process,” 9th International Conference on Reuse of Off-the-Shelf Components, pp. 395-398.
Caldeira, G., Rombach, H. and Basili, V. (1994) “Goal Question Metric Paradigm,” John Wiley Sons, vol. 1.
Cunha, R., Conte, T., Almeida, E. and Maldonado, J. (2012) “A Set of Inspection Techniques on Software Product Line Models,“ 24th International Software Engineering & Knowledge Engineering, pp. 657-662.
de Mello, R., Teixeira, E., Schots, M., Werner, C. and Travassos, G. (2012) “Checklist based inspection technique for feature models review,” in Sixth Brazilian Symposium on Software Components Architectures and Reuse, pp. 140–149.
de Mello, R. M., Teixeira, E. N., Schots, M., Werner, C. M. L. and Travassos, G. H. (2014) “Verification of software product line artifacts: A checklist to support feature model inspections,” Jornal of Universal Comp. Science, vol. 20, no. 5, pp. 720–745.
de Mello, R. M., Stolee, K. T. and Travassos, G. H. (2015) “Investigating Samples Representativeness for an Online Experiment in Java Code Search” in Nineth ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 1-10.
Gómez, O., Juristo, N. and Vegas, S. (2014) "Understanding replication of experiments in software engineering: A classification," Info and Sw Technology 56.8: 1033-1048.
Kang, K., Cohen, S., Hess, J., Nowak, W. and Peterson, S. (1990) “Feature-Oriented Domain Analysis Feasibility Study,” Technical Report CMU/SEI-90-TR-21.
Northrop, L. M. (2002) “SEI's software product line tenets,” IEEE software, vol 19, no. 4, pp. 32-40.
Rocha, A. R. C., Maldonado, J. C., Weber, K. C. and Travassos, G. H. (2001) “Qualidade de Software Teoria e Prática,” (in Portuguese) Prentice Hall.
Shull, F., Rus, I. and Basili, V. (2000) “How perspective-based reading can improve requirements inspections,” Computer, vol. 33, no. 7, pp. 73–79.
Souza, I. S., Gomes, G. S. S., Silveira, P. A. M., Machado, I. C., Almeida, E. S. and Meira, S. R. L. (2013) “Evidence of software inspection on feature specification for software product lines,” The Journal of Systems and Software 86: 1172– 1190.
Von Der Massen, T. and Lichter, H. (2004) “Deficiencies in feature models,” Workshop on Software Variability Management for Product Derivation Towards Tool Support.
Zhang, G., Ye, H. and Lin, Y. (2013) “An approach for validating feature models in software product lines,” Journal of Software Engineering, vol. 7, pp. 1-29
Benavides, D., Segura, S., Trinidad, P. and Ruiz-Cortés, A. (2007) “Fama: Tooling a framework for the automated analysis of feature models,” First International Workshop on Variability Modeling of Software-Intensive Systems, pp. 129-134.
Benavides, D., Segura, S. and Ruiz-Cortés, A. (2010) “Automated analysis of feature models 20 years later: A literature review,” Info System, vol. 35, no. 6, pp. 615-636.
Blois, A. P. T. B., de Oliveira, R. F., Maia, N., Werner, C. and Becker, K. (2006) “Variability modeling in a component-based domain engineering process,” 9th International Conference on Reuse of Off-the-Shelf Components, pp. 395-398.
Caldeira, G., Rombach, H. and Basili, V. (1994) “Goal Question Metric Paradigm,” John Wiley Sons, vol. 1.
Cunha, R., Conte, T., Almeida, E. and Maldonado, J. (2012) “A Set of Inspection Techniques on Software Product Line Models,“ 24th International Software Engineering & Knowledge Engineering, pp. 657-662.
de Mello, R., Teixeira, E., Schots, M., Werner, C. and Travassos, G. (2012) “Checklist based inspection technique for feature models review,” in Sixth Brazilian Symposium on Software Components Architectures and Reuse, pp. 140–149.
de Mello, R. M., Teixeira, E. N., Schots, M., Werner, C. M. L. and Travassos, G. H. (2014) “Verification of software product line artifacts: A checklist to support feature model inspections,” Jornal of Universal Comp. Science, vol. 20, no. 5, pp. 720–745.
de Mello, R. M., Stolee, K. T. and Travassos, G. H. (2015) “Investigating Samples Representativeness for an Online Experiment in Java Code Search” in Nineth ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 1-10.
Gómez, O., Juristo, N. and Vegas, S. (2014) "Understanding replication of experiments in software engineering: A classification," Info and Sw Technology 56.8: 1033-1048.
Kang, K., Cohen, S., Hess, J., Nowak, W. and Peterson, S. (1990) “Feature-Oriented Domain Analysis Feasibility Study,” Technical Report CMU/SEI-90-TR-21.
Northrop, L. M. (2002) “SEI's software product line tenets,” IEEE software, vol 19, no. 4, pp. 32-40.
Rocha, A. R. C., Maldonado, J. C., Weber, K. C. and Travassos, G. H. (2001) “Qualidade de Software Teoria e Prática,” (in Portuguese) Prentice Hall.
Shull, F., Rus, I. and Basili, V. (2000) “How perspective-based reading can improve requirements inspections,” Computer, vol. 33, no. 7, pp. 73–79.
Souza, I. S., Gomes, G. S. S., Silveira, P. A. M., Machado, I. C., Almeida, E. S. and Meira, S. R. L. (2013) “Evidence of software inspection on feature specification for software product lines,” The Journal of Systems and Software 86: 1172– 1190.
Von Der Massen, T. and Lichter, H. (2004) “Deficiencies in feature models,” Workshop on Software Variability Management for Product Derivation Towards Tool Support.
Zhang, G., Ye, H. and Lin, Y. (2013) “An approach for validating feature models in software product lines,” Journal of Software Engineering, vol. 7, pp. 1-29
Publicado
24/10/2016
Como Citar
SOUZA, Iuri S.; DE MELLO, Rafael M.; ALMEIDA, Eduardo S. de; WERNER, Cláudia M. L.; TRAVASSOS, Guilherme H..
Experimental Evaluation of FMCheck: A Replication Study. In: SIMPÓSIO BRASILEIRO DE QUALIDADE DE SOFTWARE (SBQS), 15. , 2016, Maceió.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2016
.
p. 121-135.
DOI: https://doi.org/10.5753/sbqs.2016.15130.
