Achievements, Challenges and Opportunities on Mutation Testing of Concurrent Programs
Resumo
With the increasing advances in the hardware technology and the massive presence of multicore processors in personal computers, concurrent programming has been becoming more popular. Therefore, new challenges have been emerged due to the communication and synchronization aspects inherent in concurrent programs. The testing activity in this scenario is considered essential for the delivery of reliable programs. Mutation testing is a fault-based testing criterion that uses mistakes done by developers during the software development. The application of this criterion in concurrent programs is challenging due to the non-determinism and concurrency problems, such as starvation and deadlock. Studies have been conducted for the application of the mutation test in concurrent programs, however, some aspects still need attention and improvements, leading researchers to face a collection of challenges. This paper presents a roadmap of this field identifying the achievements, challenges, and opportunities to study in the context of mutation testing of concurrent programs. As a result, researchers may find a guide for the proposal of new researches in the field.
Referências
Almasi, G. S., & Gottlieb, A. (1989). Highly parallel computing. Benjamin-Cummings Publishing Co., Inc.
Andrews, G. (2001). Foundations of Multithreaded, Parallel, and Distributed Programming. Addison-Wesley.
Bradbury, J. S., Cordy, J. R., & Dingel, J. (2006). Mutation operators for concurrent Java (J2SE 5.0). In Proceedings of the second workshop on mutation analysis (pp.11–20). IEEE.
Carver, R. (1993). Mutation-based testing of concurrent programs. In Proceedings of test conference (pp. 845–853).
Delamaro, M., Maldonado, J., Pezzè, M., & Vincenzi, A. (2001). Mutant operators fortesting concurrent Java programs. In Xv simpósio brasileiro de es.
Delamaro, M. E. (2004). Using instrumentation to reproduce the execution of Java concurrent programs. In Simpósio brasileiro de qualidade de software.
Farchi, E., Nir, Y., & Ur, S. (2003). Concurrent bug patterns and how to test them. In Proceedings of the 17th international symposium on parallel and distributed processing. IEEE.
Ghosh, S. (2002). Towards measurement of testability of concurrent object-oriented programs using fault insertion: a preliminary investigation. In Second ieee international workshop on source code analysis and manipulation (pp. 17–25).
Giacometti, C., Souza, S. R. S., & Souza, P. S. L. (2003). Teste de mutação para a validação de aplicações concorrentes usando PVM. In Revista eletrˆonica de iniciação científica (Vol. 2).
Gligoric, M., Jagannath, V., & Marinov, D. (2010). Mutmut: Efficient exploration for mutation testing of multithreaded code. In Proceedings of the 2010 third international conference on software testing, verification and validation (pp. 55–64). Washington, DC, USA: IEEE.
Gligoric, M., Zhang, L., Pereira, C., & Pokam, G. (2013). Selective mutation testing for concurrent code. In Proceedings of the 2013 international symposium on softwaretesting and analysis (pp. 224–234).
Hausen, A. C., Vergílio, S. R., Souza, S. R. S., Souza, P. S. L., & Sim˜ao, A. S. (2007). A tool for structural testing of MPI programs. In Ieee latin-american testworkshop - latw. IEEE.
Hunter, C., & Strooper, P. (2001). Systematically deriving partial oracles for testing concurrent programs. In Proceedings 24th australian computer science conference. acsc 2001 (p. 83-91).
Jagannath, V., Gligoric, M., Lauterburg, S., Marinov, D., & Agha, G. (2010). Mutation operators for actor systems. In Proceedings of the 2010 third international conference on software testing, verification, and validation workshops (pp. 157–162).
Kusano, M., & Wang, C. (2013). Ccmutator: A mutation generator for concurrency constructs in multithreaded c/c++ applications. In Ase (p. 722-725).
Lei, J., & Carver, R. H. (2010). A stateful approach to testing monitors in multithreaded programs. 9th IEEE International Symposium on High-Assurance Systems Engineering, 00, 54-63.
Melo, S. M., Souza, S. R. S., Silva, R. A., & Souza, P. (2015). Concurrent software testing in practice: A catalog of tools. In 6th international workshop on automating test case design, selection and evaluation (pp. 31–40).
Offutt, A. J., Voas, J., & Payne, J. (1996). Mutation operators for Ada (Tech. Rep.). George Mason University.
Schuler, D., & Zeller, A. (2009). Javalanche: Efficient mutation testing for java. In Esec/fse ’09 (pp. 297–298). New York, NY, USA: ACM.
Silva, R. A. (2013). Mutation testing applied to concurrent programs in MPI (MSc Thesis). ICMC, Instituto de Computação e Matemática Computacional, USP.
Silva, R. A. (2018). Search based software testing for the generation of synchronization sequences for mutation testing of concurrent programs (PhD Thesis). Institute of Mathematics and Computer Sciences University of S˜ao Paulo.
Silva, R. A., Souza, S. R. S., & Souza, P. S. L. (2012a). Deterministic execution of concurrent programs during the mutation testing. In 6th brazilian workshop on systematic and automated software testing.
Silva, R. A., Souza, S. R. S., & Souza, P. S. L. (2012b). Mutation testing for concurrent programs in MPI. In 13th latin american test workshop (pp. 69–74).
Silva-Barradas, S. (1998). Mutation analysis of concurrent software (PhD Dissertation). Dottorato di Ricerca in Ingegneria Informatica e Automatica, Poli. di Milano. SIR. (2018). Software-artifact infrastructure repository. Retrieved 2018-02-20, from http://sir.unl.edu
Souza, S. R. S., Brito, M. A. S., Silva, R. A., Souza, P. S. L., & Zaluska, E. (2011). Research in concurrent software testing: A systematic review. In Proceedings of the workshop on parallel and distributed systems: Testing, analysis, and debugging.
Souza, S. R. S., Souza, P. S. L., Brito, M. A. S., Simao, A. S.,&Zaluska, E. J. (2015). Empirical evaluation of a new composite approach to the coverage criteria and reachability testing of concurrent programs. Softw. Test. Verif. Reliab., 25(3), 310–332.
Tai, K., Carver, R., & Obaid, E. (1989). Deterministic execution debugging of concurrent Ada programs. In Proceedings of the computer software and applications conference (pp. 102–109).
Wu, G., & Kaiser, L. (2011). Constructing subtle concurrency bugs using synchronization-centric second-order mutation operators. In Seke (p. 244-249).
Andrews, G. (2001). Foundations of Multithreaded, Parallel, and Distributed Programming. Addison-Wesley.
Bradbury, J. S., Cordy, J. R., & Dingel, J. (2006). Mutation operators for concurrent Java (J2SE 5.0). In Proceedings of the second workshop on mutation analysis (pp.11–20). IEEE.
Carver, R. (1993). Mutation-based testing of concurrent programs. In Proceedings of test conference (pp. 845–853).
Delamaro, M., Maldonado, J., Pezzè, M., & Vincenzi, A. (2001). Mutant operators fortesting concurrent Java programs. In Xv simpósio brasileiro de es.
Delamaro, M. E. (2004). Using instrumentation to reproduce the execution of Java concurrent programs. In Simpósio brasileiro de qualidade de software.
Farchi, E., Nir, Y., & Ur, S. (2003). Concurrent bug patterns and how to test them. In Proceedings of the 17th international symposium on parallel and distributed processing. IEEE.
Ghosh, S. (2002). Towards measurement of testability of concurrent object-oriented programs using fault insertion: a preliminary investigation. In Second ieee international workshop on source code analysis and manipulation (pp. 17–25).
Giacometti, C., Souza, S. R. S., & Souza, P. S. L. (2003). Teste de mutação para a validação de aplicações concorrentes usando PVM. In Revista eletrˆonica de iniciação científica (Vol. 2).
Gligoric, M., Jagannath, V., & Marinov, D. (2010). Mutmut: Efficient exploration for mutation testing of multithreaded code. In Proceedings of the 2010 third international conference on software testing, verification and validation (pp. 55–64). Washington, DC, USA: IEEE.
Gligoric, M., Zhang, L., Pereira, C., & Pokam, G. (2013). Selective mutation testing for concurrent code. In Proceedings of the 2013 international symposium on softwaretesting and analysis (pp. 224–234).
Hausen, A. C., Vergílio, S. R., Souza, S. R. S., Souza, P. S. L., & Sim˜ao, A. S. (2007). A tool for structural testing of MPI programs. In Ieee latin-american testworkshop - latw. IEEE.
Hunter, C., & Strooper, P. (2001). Systematically deriving partial oracles for testing concurrent programs. In Proceedings 24th australian computer science conference. acsc 2001 (p. 83-91).
Jagannath, V., Gligoric, M., Lauterburg, S., Marinov, D., & Agha, G. (2010). Mutation operators for actor systems. In Proceedings of the 2010 third international conference on software testing, verification, and validation workshops (pp. 157–162).
Kusano, M., & Wang, C. (2013). Ccmutator: A mutation generator for concurrency constructs in multithreaded c/c++ applications. In Ase (p. 722-725).
Lei, J., & Carver, R. H. (2010). A stateful approach to testing monitors in multithreaded programs. 9th IEEE International Symposium on High-Assurance Systems Engineering, 00, 54-63.
Melo, S. M., Souza, S. R. S., Silva, R. A., & Souza, P. (2015). Concurrent software testing in practice: A catalog of tools. In 6th international workshop on automating test case design, selection and evaluation (pp. 31–40).
Offutt, A. J., Voas, J., & Payne, J. (1996). Mutation operators for Ada (Tech. Rep.). George Mason University.
Schuler, D., & Zeller, A. (2009). Javalanche: Efficient mutation testing for java. In Esec/fse ’09 (pp. 297–298). New York, NY, USA: ACM.
Silva, R. A. (2013). Mutation testing applied to concurrent programs in MPI (MSc Thesis). ICMC, Instituto de Computação e Matemática Computacional, USP.
Silva, R. A. (2018). Search based software testing for the generation of synchronization sequences for mutation testing of concurrent programs (PhD Thesis). Institute of Mathematics and Computer Sciences University of S˜ao Paulo.
Silva, R. A., Souza, S. R. S., & Souza, P. S. L. (2012a). Deterministic execution of concurrent programs during the mutation testing. In 6th brazilian workshop on systematic and automated software testing.
Silva, R. A., Souza, S. R. S., & Souza, P. S. L. (2012b). Mutation testing for concurrent programs in MPI. In 13th latin american test workshop (pp. 69–74).
Silva-Barradas, S. (1998). Mutation analysis of concurrent software (PhD Dissertation). Dottorato di Ricerca in Ingegneria Informatica e Automatica, Poli. di Milano. SIR. (2018). Software-artifact infrastructure repository. Retrieved 2018-02-20, from http://sir.unl.edu
Souza, S. R. S., Brito, M. A. S., Silva, R. A., Souza, P. S. L., & Zaluska, E. (2011). Research in concurrent software testing: A systematic review. In Proceedings of the workshop on parallel and distributed systems: Testing, analysis, and debugging.
Souza, S. R. S., Souza, P. S. L., Brito, M. A. S., Simao, A. S.,&Zaluska, E. J. (2015). Empirical evaluation of a new composite approach to the coverage criteria and reachability testing of concurrent programs. Softw. Test. Verif. Reliab., 25(3), 310–332.
Tai, K., Carver, R., & Obaid, E. (1989). Deterministic execution debugging of concurrent Ada programs. In Proceedings of the computer software and applications conference (pp. 102–109).
Wu, G., & Kaiser, L. (2011). Constructing subtle concurrency bugs using synchronization-centric second-order mutation operators. In Seke (p. 244-249).
Publicado
22/10/2018
Como Citar
SILVA, Rodolfo; DE SOUZA, Simone.
Achievements, Challenges and Opportunities on Mutation Testing of Concurrent Programs. In: ESCOLA REGIONAL DE ENGENHARIA DE SOFTWARE (ERES), 2. , 2018, Dois Vizinhos/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2018
.
p. 137-146.
