Is there time for Software Testing in the Indie Games Development?: A survey with practitioners of the game industry
Resumo
In 2018, the game industry made about 137,9 billion dollars worldwide in a very competitive market, since each day new games are launched. Games can be created by large development companies; or by small teams with few resources, which pursue innovation and originality in games, commonly referred to as indie games. Regardless of the size of the organization and/or team, the complexity of the development related to the production of such software is similar. In order to deliver high quality games, the practitioners carry out software tests during the production of their indie games. These tests aim to analyze the specific characteristics of this type of software, such as the degree of entertainment and playability aspects. However, there still no consensus about how the test activities are performed during the development process of indie games. This paper presents a survey to identify the test practices during the development of indie games, aiming to understand how these activities are performed. As a result, we identified that developers worry about the execution of tests during the development of indie games, focusing on techniques that consider more than one inherent aspects of the game. Within the aspects that these developers focus on, we can name: the identification of bugs, verification of playability aspects and degree of entertainment of the game.
Palavras-chave:
Survey, Software Testing, Indie Games Development
Referências
Saiqa Aleem, Luiz Fernando Capretz, and Faheem Ahmed. 2016. Critical success factors to improve the game development process from a developer's perspective. Journal of Computer Science and Technology 31, 5 (2016), 925--950.
Saiqa Aleem, Luiz Fernando Capretz, and Faheem Ahmed. 2016. Game development software engineering process life cycle: a systematic review. Journal of Software Engineering Research and Development 4, 1 (2016), 6.
Apostolos Ampatzoglou and Ioannis Stamelos. 2010. Software engineering research for computer games: A systematic review. Information and Software Technology 52, 9 (2010), 888--901.
David Callele, Eric Neufeld, and Kevin Schneider. 2005. Requirements engineering and the creative process in the video game industry. In 13th IEEE International Conference on Requirements Engineering (RE'05). IEEE, 240--250.
Heather Maxwell Chandler. 2009. The Game Production Handbook. Jones & Bartlett Publishers, MA, USA.
Judeth Oden Choi, Jodi Forlizzi, Michael Christel, Rachel Moeller, MacKenzie Bates, and Jessica Hammer. 2016. Playtesting with a Purpose. In Proceedings of the 2016 annual symposium on computer-human interaction in play. ACM, 254--265.
Juliet Corbin, Anselm Strauss, et al. 2008. Basics of qualitative research: Techniques and procedures for developing grounded theory. (2008).
Bharath T S Divya, CD. 2016. A Brief Overview of Software Testing Techniques and Test Automation. In Proceedings of the International Journal For Technological Research In Engineering, Vol. 3. 1002--1006.
Damien Djaouti, Julian Alvarez, Jean-Pierre Jessel, Gilles Methel, and Pierre Molinier. 2008. A gameplay definition through videogame classification. International Journal of Computer Games Technology 2008 (2008), 4.
Mirjam P Eladhari and Elina MI Ollila. 2012. Design for research results: experimental prototyping and play testing. Simulation & Gaming 43, 3 (2012), 391--412.
Maria B Garda and Paweł Grabarczyk. 2016. Is every indie game independent? Towards the concept of independent game. Game Studies 16, 1 (2016).
Ahmad Nauman Ghazi, Kai Petersen, Sri Sai Vijay Raj Reddy, and Harini Nekkanti. 2019. Survey Research in Software Engineering: Problems and Mitigation Strategies. IEEE Access 7 (2019), 24703--24718.
Barney G Glaser, Anselm L Strauss, and Elizabeth Strutzel. 1968. The discovery of grounded theory; strategies for qualitative research. Nursing research 17, 4 (1968), 364.
André Godoy and Ellen F Barbosa. 2010. Game-Scrum: An approach to agile game development. Proceedings of SBGames (2010), 292--295.
Christopher M Kanode and Hisham M Haddad. 2009. Software engineering challenges in game development. In 2009 Sixth International Conference on Information Technology: New Generations. IEEE, 260--265.
Jussi Kasurinen, Andrey Maglyas, and Kari Smolander. 2014. Is requirements engineering useless in game development?. In International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, 1--16.
J. Kasurinen, M. Palacin-Silva, and E. Vanhala. 2017. What Concerns Game Developers? A Study on Game Development Processes, Sustainability and Metrics. In 2017 IEEE/ACM 8th Workshop on Emerging Trends in Software Metrics (WETSoM). 15--21. https://doi.org/10.1109/WETSoM.2017.3
Jussi Kasurinen and Kari Smolander. 2014. What do game developers test in their products? International Symposium on Empirical Software Engineering and Measurement (09 2014). https://doi.org/10.1145/2652524.2652525
Clinton Keith. 2010. Agile game development with Scrum. Pearson Education.
Hannu Korhonen. 2010. Comparison of playtesting and expert review methods in mobile game evaluation. In Proceedings of the 3rd International Conference on Fun and Games. ACM, 18--27.
Jussi Koutonen and Mauri Leppänen. 2013. How are agile methods and practices deployed in video game development? A survey into Finnish game studios. In International Conference on Agile Software Development. Springer, 135--149.
A Laubisch and E Clua. 2010. Scrum4Games: Uma aplicação do Scrum para projetos de games focada em game design. Simpósio Brasileiro de Jogos e Entretenimento Digital (2010), 178--187.
Luis Levy and Jeannie Novak. 2009. Game development essentials: Game QA & testing. Cengage Learning.
Chris Lewis, Jim Whitehead, and Noah Wardrip-Fruin. 2010. What went wrong: a taxonomy of video game bugs. In Proceedings of the fifth international conference on the foundations of digital games. ACM, 108--115.
Nadav Lipkin. 2013. Examining Indie's Independence: The meaning of" Indie" Games, the politics of production, and mainstream cooptation. Loading... 7, 11 (2013).
Pejman Mirza-Babaei, Naeem Moosajee, and Brandon Drenikow. 2016. Playtesting for indie studios. In Proceedings of the 20th International Academic Mindtrek Conference. ACM, 366--374.
Emerson Murphy-Hill, Thomas Zimmermann, and Nachiappan Nagappan. 2014. Cowboys, Ankle Sprains, and Keepers of Quality: How is Video Game Development Different from Software Development?. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 1--11. https://doi.org/10.1145/2568225.2568226
Ann Osborne O'Hagan and Rory V O'Connor. 2015. Towards an Understanding of Game Software Development Processes: A Case Study. In European Conference on Software Process Improvement. Springer, 3--16.
Felan Parker. 2013. Indie Game Studies Year Eleven.. In DiGRA Conference.
Leônidas Pereira and Mauricio Bernardes. 2018. Aspects of Independent Game Production: An Exploratory Study. Computers in Entertainment 16 (11 2018), 1--16. https://doi.org/10.1145/3276322
Fábio Petrillo, Marcelo Pimenta, Francisco Trindade, and Carlos Dietrich. 2009. What went wrong? A survey of problems in game development. Computers in Entertainment (CIE) 7, 1 (2009), 13.
Cristiano Politowski, Daniel de Vargas, Lisandra M Fontoura, and Antônio A Foletto. 2016. Software Engineering Processes in Game Development: a Survey about Brazilian Developers' Experiences. Proceedings of the XIV Brazilian Symposium on Computer Games and Digital Entertainment (SBGames) (2016).
Ariel S. Posvolski, Igor V. Torres, Ilana A. Souza-Concilio, and Beatriz A. Pacheco. 2014. AgiGame: Proposta de uma Metodologia Híbrida para Desenvolvimento de Jogos. Proceedings of the XIII Brazilian Symposium on Computer Games and Digital Entertainment (SBGames) (2014).
Luiz Ojima Sakuda and Ivelise (Orgs.) Fortim. 2018. II Censo da Indústria Brasileira de Jogos Digitais. Ministério da Cultura: Brasília (2018). http://www.tinyurl.com/censojogosdigitais
Rafael Sales. 2013. Proposta de método para gestão ágil da visão no desenvolvimento de jogos digitais. Anais do XII Simpósio Brasileiro de Jogos e Entretenimento Digital (2013).
Charles P. Schultz and Robert Denton Bryant. 2016. Game Testing: All in One (3rd ed.). Mercury Learning & Information, USA.
Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. 2016. Grounded theory in software engineering research: a critical review and guidelines. In Software Engineering (ICSE), 2016 IEEE/ACM 38th International Conference on. IEEE, 120--131.
Anselm Strauss and Juliet M Corbin. 1997. Grounded theory in practice. Sage.
V Vicente Filho and Geber L Ramalho. 2014. Deepening the understanding of mobile game success. Proceedings of the XIII Brazilian Symposium on Computer Games and Digital Entertainment (SBGames) (2014).
Brian Winn and Carrie Heeter. 2006. Resolving conflicts in educational game design through playtesting. Innovate: Journal of Online Education 3, 2 (2006).
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in Software Engineering. Springer Science & Business Media.
Pedro Santoro Zambon. 2017. As vantagens de ser independente: inovação e criatividade na indústria brasileira de jogos digitais e suas influências no processo produtivo. Metamorfose 2, 1 (2017).
Samer Zein, Norsaremah Salleh, and John Grundy. 2016. A systematic mapping study of mobile application testing techniques. Journal of Systems and Software 117 (2016), 334--356.
Saiqa Aleem, Luiz Fernando Capretz, and Faheem Ahmed. 2016. Game development software engineering process life cycle: a systematic review. Journal of Software Engineering Research and Development 4, 1 (2016), 6.
Apostolos Ampatzoglou and Ioannis Stamelos. 2010. Software engineering research for computer games: A systematic review. Information and Software Technology 52, 9 (2010), 888--901.
David Callele, Eric Neufeld, and Kevin Schneider. 2005. Requirements engineering and the creative process in the video game industry. In 13th IEEE International Conference on Requirements Engineering (RE'05). IEEE, 240--250.
Heather Maxwell Chandler. 2009. The Game Production Handbook. Jones & Bartlett Publishers, MA, USA.
Judeth Oden Choi, Jodi Forlizzi, Michael Christel, Rachel Moeller, MacKenzie Bates, and Jessica Hammer. 2016. Playtesting with a Purpose. In Proceedings of the 2016 annual symposium on computer-human interaction in play. ACM, 254--265.
Juliet Corbin, Anselm Strauss, et al. 2008. Basics of qualitative research: Techniques and procedures for developing grounded theory. (2008).
Bharath T S Divya, CD. 2016. A Brief Overview of Software Testing Techniques and Test Automation. In Proceedings of the International Journal For Technological Research In Engineering, Vol. 3. 1002--1006.
Damien Djaouti, Julian Alvarez, Jean-Pierre Jessel, Gilles Methel, and Pierre Molinier. 2008. A gameplay definition through videogame classification. International Journal of Computer Games Technology 2008 (2008), 4.
Mirjam P Eladhari and Elina MI Ollila. 2012. Design for research results: experimental prototyping and play testing. Simulation & Gaming 43, 3 (2012), 391--412.
Maria B Garda and Paweł Grabarczyk. 2016. Is every indie game independent? Towards the concept of independent game. Game Studies 16, 1 (2016).
Ahmad Nauman Ghazi, Kai Petersen, Sri Sai Vijay Raj Reddy, and Harini Nekkanti. 2019. Survey Research in Software Engineering: Problems and Mitigation Strategies. IEEE Access 7 (2019), 24703--24718.
Barney G Glaser, Anselm L Strauss, and Elizabeth Strutzel. 1968. The discovery of grounded theory; strategies for qualitative research. Nursing research 17, 4 (1968), 364.
André Godoy and Ellen F Barbosa. 2010. Game-Scrum: An approach to agile game development. Proceedings of SBGames (2010), 292--295.
Christopher M Kanode and Hisham M Haddad. 2009. Software engineering challenges in game development. In 2009 Sixth International Conference on Information Technology: New Generations. IEEE, 260--265.
Jussi Kasurinen, Andrey Maglyas, and Kari Smolander. 2014. Is requirements engineering useless in game development?. In International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, 1--16.
J. Kasurinen, M. Palacin-Silva, and E. Vanhala. 2017. What Concerns Game Developers? A Study on Game Development Processes, Sustainability and Metrics. In 2017 IEEE/ACM 8th Workshop on Emerging Trends in Software Metrics (WETSoM). 15--21. https://doi.org/10.1109/WETSoM.2017.3
Jussi Kasurinen and Kari Smolander. 2014. What do game developers test in their products? International Symposium on Empirical Software Engineering and Measurement (09 2014). https://doi.org/10.1145/2652524.2652525
Clinton Keith. 2010. Agile game development with Scrum. Pearson Education.
Hannu Korhonen. 2010. Comparison of playtesting and expert review methods in mobile game evaluation. In Proceedings of the 3rd International Conference on Fun and Games. ACM, 18--27.
Jussi Koutonen and Mauri Leppänen. 2013. How are agile methods and practices deployed in video game development? A survey into Finnish game studios. In International Conference on Agile Software Development. Springer, 135--149.
A Laubisch and E Clua. 2010. Scrum4Games: Uma aplicação do Scrum para projetos de games focada em game design. Simpósio Brasileiro de Jogos e Entretenimento Digital (2010), 178--187.
Luis Levy and Jeannie Novak. 2009. Game development essentials: Game QA & testing. Cengage Learning.
Chris Lewis, Jim Whitehead, and Noah Wardrip-Fruin. 2010. What went wrong: a taxonomy of video game bugs. In Proceedings of the fifth international conference on the foundations of digital games. ACM, 108--115.
Nadav Lipkin. 2013. Examining Indie's Independence: The meaning of" Indie" Games, the politics of production, and mainstream cooptation. Loading... 7, 11 (2013).
Pejman Mirza-Babaei, Naeem Moosajee, and Brandon Drenikow. 2016. Playtesting for indie studios. In Proceedings of the 20th International Academic Mindtrek Conference. ACM, 366--374.
Emerson Murphy-Hill, Thomas Zimmermann, and Nachiappan Nagappan. 2014. Cowboys, Ankle Sprains, and Keepers of Quality: How is Video Game Development Different from Software Development?. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 1--11. https://doi.org/10.1145/2568225.2568226
Ann Osborne O'Hagan and Rory V O'Connor. 2015. Towards an Understanding of Game Software Development Processes: A Case Study. In European Conference on Software Process Improvement. Springer, 3--16.
Felan Parker. 2013. Indie Game Studies Year Eleven.. In DiGRA Conference.
Leônidas Pereira and Mauricio Bernardes. 2018. Aspects of Independent Game Production: An Exploratory Study. Computers in Entertainment 16 (11 2018), 1--16. https://doi.org/10.1145/3276322
Fábio Petrillo, Marcelo Pimenta, Francisco Trindade, and Carlos Dietrich. 2009. What went wrong? A survey of problems in game development. Computers in Entertainment (CIE) 7, 1 (2009), 13.
Cristiano Politowski, Daniel de Vargas, Lisandra M Fontoura, and Antônio A Foletto. 2016. Software Engineering Processes in Game Development: a Survey about Brazilian Developers' Experiences. Proceedings of the XIV Brazilian Symposium on Computer Games and Digital Entertainment (SBGames) (2016).
Ariel S. Posvolski, Igor V. Torres, Ilana A. Souza-Concilio, and Beatriz A. Pacheco. 2014. AgiGame: Proposta de uma Metodologia Híbrida para Desenvolvimento de Jogos. Proceedings of the XIII Brazilian Symposium on Computer Games and Digital Entertainment (SBGames) (2014).
Luiz Ojima Sakuda and Ivelise (Orgs.) Fortim. 2018. II Censo da Indústria Brasileira de Jogos Digitais. Ministério da Cultura: Brasília (2018). http://www.tinyurl.com/censojogosdigitais
Rafael Sales. 2013. Proposta de método para gestão ágil da visão no desenvolvimento de jogos digitais. Anais do XII Simpósio Brasileiro de Jogos e Entretenimento Digital (2013).
Charles P. Schultz and Robert Denton Bryant. 2016. Game Testing: All in One (3rd ed.). Mercury Learning & Information, USA.
Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. 2016. Grounded theory in software engineering research: a critical review and guidelines. In Software Engineering (ICSE), 2016 IEEE/ACM 38th International Conference on. IEEE, 120--131.
Anselm Strauss and Juliet M Corbin. 1997. Grounded theory in practice. Sage.
V Vicente Filho and Geber L Ramalho. 2014. Deepening the understanding of mobile game success. Proceedings of the XIII Brazilian Symposium on Computer Games and Digital Entertainment (SBGames) (2014).
Brian Winn and Carrie Heeter. 2006. Resolving conflicts in educational game design through playtesting. Innovate: Journal of Online Education 3, 2 (2006).
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in Software Engineering. Springer Science & Business Media.
Pedro Santoro Zambon. 2017. As vantagens de ser independente: inovação e criatividade na indústria brasileira de jogos digitais e suas influências no processo produtivo. Metamorfose 2, 1 (2017).
Samer Zein, Norsaremah Salleh, and John Grundy. 2016. A systematic mapping study of mobile application testing techniques. Journal of Systems and Software 117 (2016), 334--356.
Publicado
23/09/2019
Como Citar
DE OLIVEIRA NETO, José Nunes; VIANA, Davi; SÁ, Eveline; RIVERO, Luis; LOPES, Rafael Fernandes; SILVA, Francisco.
Is there time for Software Testing in the Indie Games Development?: A survey with practitioners of the game industry. In: SIMPÓSIO BRASILEIRO DE ENGENHARIA DE SOFTWARE (SBES), 33. , 2019, Salvador.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
