skip to main content
10.1145/3629479.3629496acmotherconferencesArticle/Chapter ViewAbstractPublication PagessbqsConference Proceedingsconference-collections
research-article

An Open Educational Resource Supporting Mutation Testing Teaching

Published:06 December 2023Publication History

ABSTRACT

Mutation testing is widely recognized as one of the most effective criteria for software testing to uncover defects. However, there is an urgent need for more significant efforts to promote learning and understanding of this criterion. Currently, teaching support mechanisms in software testing tend to prioritize other criteria over mutation testing, exacerbating the gap. In this paper, we introduce MUT-STT, an open educational resource specifically designed to address this gap and facilitate the dissemination of critical concepts and practical aspects of mutation testing. MUT-STT focuses on the intrinsic concepts of the mutation testing criterion, providing comprehensive support for learners. It incorporates review exercises, offers tool suggestions for test automation, and provides clear instructions on applying the mutation test criterion using Python. An evaluation was conducted to ensure the educational resource’s content was complete and correct. In this evaluation, experts in mutation testing carefully examined MUT-STT, identifying and addressing any conceptual or descriptive defects that may have persisted.

References

  1. H. Agrawal, R. A. DeMillo, R_ Hathaway, W. Hsu, W. Hsu, E. W. Krauser, R. J. Martin, A. P. Mathur, and E. Spafford. 1989. Design of mutant operators for the C programming language. Technical Report. Technical Report SERC-TR-41-P, Software Engineering Research Center, Purdue ….Google ScholarGoogle Scholar
  2. P. Ammann and J. Offutt. 2016. Introduction to software testing. Cambridge University Press.Google ScholarGoogle Scholar
  3. E.F. Barbosa and J.C. Maldonado. 2006. Establishing a Mutation Testing Educational Module based on IMA-CID. In Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006). 1–10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. E. F. Barbosa and J. C. Maldonado. 2011. IMA-CID: an integrated modeling approach for developing educational modules. Journal of the Brazilian Computer Society 17 (2011), 207–239.Google ScholarGoogle ScholarCross RefCross Ref
  5. E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi. 2001. Toward the determination of sufficient mutant operators for C. Software Testing, Verification and Reliability 11, 2 (2001), 113–136.Google ScholarGoogle ScholarCross RefCross Ref
  6. V. R. Basili, R. W. Selby, and Da. H. Hutchens. 1986. Experimentation in software engineering. IEEE Transactions on Software Engineering SE-12, 7 (1986), 733–743.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. B. A. Becker and K. Quille. 2019. 50 Years of CS1 at SIGCSE: A Review of the Evolution of Introductory Programming Education Research. In ACM Technical Symposium on Computer Science Education. 338–344.Google ScholarGoogle Scholar
  8. B. Beizer. 1995. Black-box testing: techniques for functional testing of software and systems. John Wiley & Sons, Inc.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. F. B. V. Benitti. 2018. A Methodology to Define Learning Objects Granularity: A Case Study in Software Testing. Informatics in Education 17, 1 (2018), 1–20.Google ScholarGoogle ScholarCross RefCross Ref
  10. Á. Beszédes and L. Vidács. 2016. Academic and Industrial Software Testing Conferences: Survey and Synergies. In International Conference on Software Testing, Verification and Validation Workshops. 240–249.Google ScholarGoogle Scholar
  11. B. S. Clegg, J. M. Rojas, and G. Fraser. 2017. Teaching Software Testing Concepts Using a Mutation Testing Game. In International Conference on Software Engineering. 33–36.Google ScholarGoogle Scholar
  12. H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque. 2016. PIT: A Practical Mutation Testing Tool for Java (Demo). In International Symposium on Software Testing and Analysis. 449–452.Google ScholarGoogle Scholar
  13. R. De Millo, W. M. McCracken, R. J. Martin, and J. Passafiume. 1987. Software testing and evaluation. Benjamin-Cummings Publishing Co., Inc.Google ScholarGoogle Scholar
  14. S. R. S. de Souza, J. C. Maldonado, S. C. P. Forlin Fabbri, and W. L. de Souza .1999. Mutation Testing Applied to Estelle Specifications. Software Quality Journal 8 (1999), 285–301.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. M Delamaro, M L Chaim, and J. C. Maldonado. 2018. Where Are the Minimal Mutants?. In XXXII Brazilian Symposium on Software Engineering. 190–195.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. M. Delamaro, M. Jino, and J. Maldonado. 2016. Introdução ao teste de software. Elsevier Brasil.Google ScholarGoogle Scholar
  17. M. E. Delamaro, S. A. Andrade, S. R. S. de Souza, and P. S. L. de Souza. 2021. Parallel execution of programs as a support for mutation testing: a replication study. International Journal of Software Engineering and Knowledge Engineering 31, 03 (2021), 337–380.Google ScholarGoogle ScholarCross RefCross Ref
  18. M. E. Delamaro, J. C. Maldonado, and A. M. R. Vincenzi. 2001. Proteum/IM 2.0: An Integrated Mutation Testing Environment. Springer US, Boston, MA, 91–101.Google ScholarGoogle Scholar
  19. A. Derezińska and K. Hałas. 2014. Analysis of Mutation Operators for the Python Language. In International Conference on Dependability and Complex Systems, W. Zamojski, J. Mazurkiewicz, J. Sugier, T. Walkowiak, and J. Kacprzyk (Eds.). Cham, 155–164.Google ScholarGoogle Scholar
  20. A. G. O. Fassbinder. 2018. A contribution to the process of designing for learning in Massive Open Online Courses (MOOCs). Ph. D. Dissertation. Instituto de Ciências Matemáticas e de Computação, Universidade de São Paulo, São Carlos.Google ScholarGoogle Scholar
  21. L. Fernandes, M. Ribeiro, R. Gheyi, M. Delamaro, M. Guimarães, and A. Santos. 2022. Put Your Hands In The Air! Reducing Manual Effort in Mutation Testing. In XXXVI Brazilian Symposium on Software Engineering. 198–207.Google ScholarGoogle Scholar
  22. G. Fraser. 2017. Gamification of Software Testing. In International Workshop on Automation of Software Testing. 2–7.Google ScholarGoogle Scholar
  23. G. Fraser. 2019. Code Defenders: A Mutation Testing Game. In ACM Technical Symposium on Computer Science Education. 1289.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. G. Fraser, A. Gambi, M. Kreis, and J. M. Rojas. 2019. Gamifying a Software Testing Course with Code Defenders. In ACM Technical Symposium on Computer Science Education. 571–577.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. G. Fraser, A.o Gambi, and Jo. M. Rojas. 2018. A Preliminary Report on Gamifying a Software Testing Course with the Code Defenders Testing Game. In European Conference of Software Engineering Education. 50–54.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. V. Garousi, A. Rainer, P. Lauvås, and A. Arcuri. 2020. Software-testing education: A systematic literature mapping. Journal of Systems and Software 165 (2020), 110570.Google ScholarGoogle ScholarCross RefCross Ref
  27. F. Johnson, S. McQuistin, and J. O’Donnell. 2020. Analysis of Student Misconceptions Using Python as an Introductory Programming Language. In Conference on Computing Education Practice. 1–4.Google ScholarGoogle Scholar
  28. A. Kemczinski, I. A. Costa, M. A. Wehrmeister, M. da Silva Hounsell, and A. Vahldick. 2012. Metodologia para construção de objetos de aprendizagem interativos. In Brazilian Symposium on Computers in Education, Vol. 23. 1–10.Google ScholarGoogle Scholar
  29. V. P. Lopes and G. H. Travassos. 2009. Knowledge Repository Structure of an Experimental Software Engineering Environment. In Brazilian Symposium on Software Engineering. 32–42.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Yu-Seung Ma, Yong-Rae Kwon, and J. Offutt. 2002. Inter-class mutation operators for Java. In International Symposium on Software Reliability Engineering. 352–363.Google ScholarGoogle Scholar
  31. S. M. Melo, V. X. S. Moreira, L. N. Paschoal, and S. R. S. Souza. 2020. Testing education: a survey on a global scale. In XXXIV Brazilian Symposium on Software Engineering. 554–563.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. G. J. Myers, T. Badgett, and C. Sandler. 2012. Test-Case Design. John Wiley & Sons, Ltd, Chapter 4, 41–84.Google ScholarGoogle Scholar
  33. M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. L. Traon, and M. Harman. 2019. Chapter Six - Mutation Testing Advances: An Analysis and Survey. Advances in Computers, Vol. 112. 275–378.Google ScholarGoogle Scholar
  34. L. N. Paschoal. 2019. Contribuições ao ensino de teste de software com o modelo flipped classroom e um agente conversacional. Master’s thesis. Instituto de Ciências Matemáticas e de Computação, University of São Paulo, São Carlos. https://doi.org/10.11606/D.55.2019.tde-13062019-140507Google ScholarGoogle ScholarCross RefCross Ref
  35. L. N. Paschoal and S. R. S. de Souza. 2018. A survey on software testing education in Brazil. In XVII Brazilian Symposium on Software Quality. 334–343.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. L. N. Paschoal, B. R. N. Oliveira, E. Y. Nakagawa, and S. R. S Souza. 2019. Can we use the Flipped Classroom Model to teach Black-box Testing to Computer Students?. In XVIII Brazilian Symposium on Software Quality. 158–167.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. L. N. Paschoal, M. M Oliveira, S. M. Melo, E. F. Barbosa, and S. R.S. Souza. 2020. Evaluating the impact of software testing education through the flipped classroom model in deriving test requirements. In XXXIV Brazilian Symposium on Software Engineering. 570–579.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. J. M. Prates, S. M. Melo, P. H. D. Valle, R. E. Garcia, and J. C. Maldonado. 2023. Integrating SPOCs in Software Testing Education: Evidence in Emergency Remote Courses. Informatics in Education 22, 1 (2023), 121–139.Google ScholarGoogle Scholar
  39. N. B. Ruparelia. 2010. Software Development Lifecycle Models. SIGSOFT Softw. Eng. Notes 35, 3 (2010), 8–13.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. A. M.R. Vincenzi, A. P.S. Barbosa, C. dos Santos Sousa, C. Polimanti, F. Oliveira, G. de Paula, and J. C. Maldonado. 2020. Loggi: Treinamento Localizado em Automatização de Teste de Software em Ambiente Empresarial. In XI Congresso Brasileiro de Software: Teoria e Prática. SBC, 123–126.Google ScholarGoogle Scholar
  41. David Wiley and John Levi Hilton Iii. 2018. Defining OER-enabled pedagogy. The International Review of Research in Open and Distributed Learning 19, 4 (2018).Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. An Open Educational Resource Supporting Mutation Testing Teaching

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in
      • Published in

        cover image ACM Other conferences
        SBQS '23: Proceedings of the XXII Brazilian Symposium on Software Quality
        November 2023
        391 pages
        ISBN:9798400707865
        DOI:10.1145/3629479

        Copyright © 2023 ACM

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 6 December 2023

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed limited

        Acceptance Rates

        Overall Acceptance Rate35of99submissions,35%
      • Article Metrics

        • Downloads (Last 12 months)11
        • Downloads (Last 6 weeks)4

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format .

      View HTML Format