ABSTRACT
Mutation testing is widely recognized as one of the most effective criteria for software testing to uncover defects. However, there is an urgent need for more significant efforts to promote learning and understanding of this criterion. Currently, teaching support mechanisms in software testing tend to prioritize other criteria over mutation testing, exacerbating the gap. In this paper, we introduce MUT-STT, an open educational resource specifically designed to address this gap and facilitate the dissemination of critical concepts and practical aspects of mutation testing. MUT-STT focuses on the intrinsic concepts of the mutation testing criterion, providing comprehensive support for learners. It incorporates review exercises, offers tool suggestions for test automation, and provides clear instructions on applying the mutation test criterion using Python. An evaluation was conducted to ensure the educational resource’s content was complete and correct. In this evaluation, experts in mutation testing carefully examined MUT-STT, identifying and addressing any conceptual or descriptive defects that may have persisted.
- H. Agrawal, R. A. DeMillo, R_ Hathaway, W. Hsu, W. Hsu, E. W. Krauser, R. J. Martin, A. P. Mathur, and E. Spafford. 1989. Design of mutant operators for the C programming language. Technical Report. Technical Report SERC-TR-41-P, Software Engineering Research Center, Purdue ….Google Scholar
- P. Ammann and J. Offutt. 2016. Introduction to software testing. Cambridge University Press.Google Scholar
- E.F. Barbosa and J.C. Maldonado. 2006. Establishing a Mutation Testing Educational Module based on IMA-CID. In Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006). 1–10.Google ScholarDigital Library
- E. F. Barbosa and J. C. Maldonado. 2011. IMA-CID: an integrated modeling approach for developing educational modules. Journal of the Brazilian Computer Society 17 (2011), 207–239.Google ScholarCross Ref
- E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi. 2001. Toward the determination of sufficient mutant operators for C. Software Testing, Verification and Reliability 11, 2 (2001), 113–136.Google ScholarCross Ref
- V. R. Basili, R. W. Selby, and Da. H. Hutchens. 1986. Experimentation in software engineering. IEEE Transactions on Software Engineering SE-12, 7 (1986), 733–743.Google ScholarDigital Library
- B. A. Becker and K. Quille. 2019. 50 Years of CS1 at SIGCSE: A Review of the Evolution of Introductory Programming Education Research. In ACM Technical Symposium on Computer Science Education. 338–344.Google Scholar
- B. Beizer. 1995. Black-box testing: techniques for functional testing of software and systems. John Wiley & Sons, Inc.Google ScholarDigital Library
- F. B. V. Benitti. 2018. A Methodology to Define Learning Objects Granularity: A Case Study in Software Testing. Informatics in Education 17, 1 (2018), 1–20.Google ScholarCross Ref
- Á. Beszédes and L. Vidács. 2016. Academic and Industrial Software Testing Conferences: Survey and Synergies. In International Conference on Software Testing, Verification and Validation Workshops. 240–249.Google Scholar
- B. S. Clegg, J. M. Rojas, and G. Fraser. 2017. Teaching Software Testing Concepts Using a Mutation Testing Game. In International Conference on Software Engineering. 33–36.Google Scholar
- H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque. 2016. PIT: A Practical Mutation Testing Tool for Java (Demo). In International Symposium on Software Testing and Analysis. 449–452.Google Scholar
- R. De Millo, W. M. McCracken, R. J. Martin, and J. Passafiume. 1987. Software testing and evaluation. Benjamin-Cummings Publishing Co., Inc.Google Scholar
- S. R. S. de Souza, J. C. Maldonado, S. C. P. Forlin Fabbri, and W. L. de Souza .1999. Mutation Testing Applied to Estelle Specifications. Software Quality Journal 8 (1999), 285–301.Google ScholarDigital Library
- M Delamaro, M L Chaim, and J. C. Maldonado. 2018. Where Are the Minimal Mutants?. In XXXII Brazilian Symposium on Software Engineering. 190–195.Google ScholarDigital Library
- M. Delamaro, M. Jino, and J. Maldonado. 2016. Introdução ao teste de software. Elsevier Brasil.Google Scholar
- M. E. Delamaro, S. A. Andrade, S. R. S. de Souza, and P. S. L. de Souza. 2021. Parallel execution of programs as a support for mutation testing: a replication study. International Journal of Software Engineering and Knowledge Engineering 31, 03 (2021), 337–380.Google ScholarCross Ref
- M. E. Delamaro, J. C. Maldonado, and A. M. R. Vincenzi. 2001. Proteum/IM 2.0: An Integrated Mutation Testing Environment. Springer US, Boston, MA, 91–101.Google Scholar
- A. Derezińska and K. Hałas. 2014. Analysis of Mutation Operators for the Python Language. In International Conference on Dependability and Complex Systems, W. Zamojski, J. Mazurkiewicz, J. Sugier, T. Walkowiak, and J. Kacprzyk (Eds.). Cham, 155–164.Google Scholar
- A. G. O. Fassbinder. 2018. A contribution to the process of designing for learning in Massive Open Online Courses (MOOCs). Ph. D. Dissertation. Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo, São Carlos.Google Scholar
- L. Fernandes, M. Ribeiro, R. Gheyi, M. Delamaro, M. Guimarães, and A. Santos. 2022. Put Your Hands In The Air! Reducing Manual Effort in Mutation Testing. In XXXVI Brazilian Symposium on Software Engineering. 198–207.Google Scholar
- G. Fraser. 2017. Gamification of Software Testing. In International Workshop on Automation of Software Testing. 2–7.Google Scholar
- G. Fraser. 2019. Code Defenders: A Mutation Testing Game. In ACM Technical Symposium on Computer Science Education. 1289.Google ScholarDigital Library
- G. Fraser, A. Gambi, M. Kreis, and J. M. Rojas. 2019. Gamifying a Software Testing Course with Code Defenders. In ACM Technical Symposium on Computer Science Education. 571–577.Google ScholarDigital Library
- G. Fraser, A.o Gambi, and Jo. M. Rojas. 2018. A Preliminary Report on Gamifying a Software Testing Course with the Code Defenders Testing Game. In European Conference of Software Engineering Education. 50–54.Google ScholarDigital Library
- V. Garousi, A. Rainer, P. Lauvås, and A. Arcuri. 2020. Software-testing education: A systematic literature mapping. Journal of Systems and Software 165 (2020), 110570.Google ScholarCross Ref
- F. Johnson, S. McQuistin, and J. O’Donnell. 2020. Analysis of Student Misconceptions Using Python as an Introductory Programming Language. In Conference on Computing Education Practice. 1–4.Google Scholar
- A. Kemczinski, I. A. Costa, M. A. Wehrmeister, M. da Silva Hounsell, and A. Vahldick. 2012. Metodologia para construção de objetos de aprendizagem interativos. In Brazilian Symposium on Computers in Education, Vol. 23. 1–10.Google Scholar
- V. P. Lopes and G. H. Travassos. 2009. Knowledge Repository Structure of an Experimental Software Engineering Environment. In Brazilian Symposium on Software Engineering. 32–42.Google ScholarDigital Library
- Yu-Seung Ma, Yong-Rae Kwon, and J. Offutt. 2002. Inter-class mutation operators for Java. In International Symposium on Software Reliability Engineering. 352–363.Google Scholar
- S. M. Melo, V. X. S. Moreira, L. N. Paschoal, and S. R. S. Souza. 2020. Testing education: a survey on a global scale. In XXXIV Brazilian Symposium on Software Engineering. 554–563.Google ScholarDigital Library
- G. J. Myers, T. Badgett, and C. Sandler. 2012. Test-Case Design. John Wiley & Sons, Ltd, Chapter 4, 41–84.Google Scholar
- M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. L. Traon, and M. Harman. 2019. Chapter Six - Mutation Testing Advances: An Analysis and Survey. Advances in Computers, Vol. 112. 275–378.Google Scholar
- L. N. Paschoal. 2019. Contribuições ao ensino de teste de software com o modelo flipped classroom e um agente conversacional. Master’s thesis. Instituto de Ciências Matemáticas e de Computação, University of São Paulo, São Carlos. https://doi.org/10.11606/D.55.2019.tde-13062019-140507Google ScholarCross Ref
- L. N. Paschoal and S. R. S. de Souza. 2018. A survey on software testing education in Brazil. In XVII Brazilian Symposium on Software Quality. 334–343.Google ScholarDigital Library
- L. N. Paschoal, B. R. N. Oliveira, E. Y. Nakagawa, and S. R. S Souza. 2019. Can we use the Flipped Classroom Model to teach Black-box Testing to Computer Students?. In XVIII Brazilian Symposium on Software Quality. 158–167.Google ScholarDigital Library
- L. N. Paschoal, M. M Oliveira, S. M. Melo, E. F. Barbosa, and S. R.S. Souza. 2020. Evaluating the impact of software testing education through the flipped classroom model in deriving test requirements. In XXXIV Brazilian Symposium on Software Engineering. 570–579.Google ScholarDigital Library
- J. M. Prates, S. M. Melo, P. H. D. Valle, R. E. Garcia, and J. C. Maldonado. 2023. Integrating SPOCs in Software Testing Education: Evidence in Emergency Remote Courses. Informatics in Education 22, 1 (2023), 121–139.Google Scholar
- N. B. Ruparelia. 2010. Software Development Lifecycle Models. SIGSOFT Softw. Eng. Notes 35, 3 (2010), 8–13.Google ScholarDigital Library
- A. M.R. Vincenzi, A. P.S. Barbosa, C. dos Santos Sousa, C. Polimanti, F. Oliveira, G. de Paula, and J. C. Maldonado. 2020. Loggi: Treinamento Localizado em Automatização de Teste de Software em Ambiente Empresarial. In XI Congresso Brasileiro de Software: Teoria e Prática. SBC, 123–126.Google Scholar
- David Wiley and John Levi Hilton Iii. 2018. Defining OER-enabled pedagogy. The International Review of Research in Open and Distributed Learning 19, 4 (2018).Google ScholarCross Ref
Index Terms
- An Open Educational Resource Supporting Mutation Testing Teaching
Recommendations
Teaching software testing concepts using a mutation testing game
ICSE-SEET '17: Proceedings of the 39th International Conference on Software Engineering: Software Engineering and Education TrackSoftware testing is a core aspect of software development, but testing programs systematically is not always a core aspect of software engineering education. As a result, software developers often treat testing as a liability, and overall software ...
Automatic Test Generation using Checkpoint Encoding and Antirandom Testing
ISSRE '97: Proceedings of the Eighth International Symposium on Software Reliability EngineeringThe implementation of an efficient automatic test generation scheme for black-box testing is discussed. It uses checkpoint encoding and antirandom testing schemes. Checkpoint encoding converts test generation to a binary problem. The checkpoints are ...
Predictive mutation testing
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and AnalysisMutation testing is a powerful methodology for evaluating test suite quality. In mutation testing, a large number of mutants are generated and executed against the test suite to check the ratio of killed mutants. Therefore, mutation testing is widely ...
Comments