A Lightweight Technique to Identify Equivalent Mutants

• Beatriz Souza Universidade Federal de Campina Grande
• Rohit Gheyi Universidade Federal de Campina Grande

DOI: https://doi.org/10.5753/cbsoft_estendido.2020.14630

Resumo

Mutation analysis is a popular but costly approach to assess the quality of test suites. Equivalent mutants are useless and contribute to increase costs. We propose a lightweight technique to identify equivalent mutants by proving equivalences with Z3 in the context of weak mutation testing. To evaluate our approach, we apply our technique for 40 mutation targets (mutations of an expression or statement) and automatically identify 13 equivalent mutations for seven mutation targets. We manually confirm that the equivalent mutants detected by our technique are indeed equivalent. Moreover, we evaluate our approach in the context of strong mutation testing against mutants generated by MuJava for 5 projects. Our technique detects all equivalent mutants detected by TCE. The results of our technique can be useful to improve mutation testing tools by avoiding the application of 13 mutations for 7 mutation targets.

Palavras-chave: Mutation Testing, Equivalent Mutants, Theorem Proving

Referências

Acree, J. A. T. (1980). On mutation. PhD thesis, Georgia Institute of Technology.Artifacts (2020).

A technique to identify equivalent mutants. At https://colab.research.google.com/drive/1ts_2m0tW0GLWsVJ_qhC59V9l8rVPdtsa?usp=sharing.

Baldwin, D. and Sayward, F. (1979). Heuristics for determining equivalence of program mutations. Technical report, DTIC Document.

Budd, T. and Angluin, D. (1982). Two notions of correctness and their relation to testing. Acta Informatica, 18(1):31–45.

de Moura, L. M. and Bjørner, N. (2008). Z3: an efficient SMT solver. In Proceedings of the Tools and Algorithms for the Construction and Analysis of Systems, pages 337–340.

DeMillo, R. A., Lipton, R. J., and Sayward, F. G. (1978). Hints on test data selection: Help for the practicing programmer. Computer, 11(4):34–41.

Fernandes, L., Ribeiro, M., Carvalho, L., Gheyi, R., Mongiovi, M., Santos, A., Cavalcanti, A., Ferrari, F., and Maldonado, J. C. (2017). Avoiding useless mutants. In Proceedings of the Generative Programming: Concepts & Experiences, pages 187–198.

canti, A., Ferrari, F., and Maldonado, J. C. (2017). Avoiding useless mutants. In

Grün, B. J., Schuler, D., and Zeller, A. (2009). The impact of equivalent mutants. In Proceedings of the International Conference on Software Testing, Verification, and Validation Workshops, pages 192–199.

Guimarães, M., Fernandes, L., Ribeiro, M., d’Amorim, M., and Gheyi, R. (2020). Optimizing mutation testing by discovering dynamic mutant subsumption relations. In Proceedings of the International Conference on Software Testing, Verification and Validation, page To appear. IEEE.

Harman, M., Hierons, R., and Danicic, S. (2001). The relationship between program dependence and mutation analysis. In Proceedings of the Mutation Testing for the New Century, pages 5–13.

Howden, W. (1982). Weak mutation testing and completeness of test sets. Transactions on Software Engineering, 8(4):371–379.

Jia, Y. and Harman, M. (2009). Higher order mutation testing. Information and Software Technology, 51(10):1379 – 1393.

Jia, Y. and Harman, M. (2011). An analysis and survey of the development of mutation testing. Transactions on Software Engineering, 37(5):649–678.

Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., and Fraser, G. (2014). Are mutants a valid substitute for real faults in software testing? In Proceedings of the Foundations of Software Engineering, pages 654–665.

Kintis, M. and Malevris, N. (2015). MEDIC: A static analysis framework for equivalent mutant identification. Information and Software Technology, 68:1–17.

Kintis, M., Papadakis, M., Jia, Y., Malevris, N., Traon, Y. L., and Harman, M. (2017). Detecting trivial mutant equivalences via compiler optimisations. Transactions on Software Engineering, 44(4):308–333.

Ma, Y.-S. and Offutt, J. Description of MuJava’s method-level mutation operators. At https://cs.gmu.edu/offutt/mujava/mutopsMethod.pdf

Ma, Y.-S., Offutt, J., and Kwon, Y.-R. (2005). MuJava: an automated class mutation system. Software Testing, Verification and Reliability, 15(2):97–133.

Madeyski, L., Orzeszyna, W., Torkar, R., and Jozala, M. (2014). Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation. Transactions on Software Engineering, 40(1):23–42.

Offutt, A. J. and Lee, S. D. (1994). An empirical evaluation of weak mutation. Transactions on Software Engineering, 20(5):337–344.

Offutt, J. (2011). A mutation carol: Past, present and future. Information and Software Technology, 53(10):1098 – 1107.

Offutt, J. and Pan, J. (1997). Automatically detecting equivalent mutants and infeasible paths. Software Testing, Verification and Reliability, 7(3):165–192.

Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y. L., and Harman, M. (2019). Chapter six - mutation testing advances: An analysis and survey. Advances in Computers, 112:275–378.

Petrovic, G. and Ivankovic, M. (2018). State of mutation testing at google. In 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pages 163–171.

Pizzoleto, A., Ferrari, F., Offutt, J., Fernandes, L., and Ribeiro, M. (2019). A systematic literature review of techniques and metrics to reduce the cost of mutation testing. Journal of Systems and Software, 157.

Schuler, D. and Zeller, A. (2013). Covering and uncovering equivalent mutants. Software Testing, Verification and Reliability, 23(5):353–374.

Winterer, D., Zhang, C., and Su, Z. (2020). Validating SMT solvers via semantic fusion. In Proceedings of the Programming Language Design and Implementation, pages 718–730.