Uma avaliação do potencial de detecção de defeitos dos casos de teste de robustez de acordo com a análise de mutantes
Abstract
Testing whether a system satisfies the specified functionalities is necessary, but not enough. It is also important to determine how a system behave in presence of invalid or unexpected inputs. This is the goal of robustness testing. If robustness faults go undetected during development, they can occur when the system is in operation, when faults are harder to diagnose and fix, which can lead to total interruption of system operations. In this paper we investigate the fault detection potential of nominal and robustness test sequences. Nominal sequences consider only specified inputs, whereas the robustness ones consider also innoportune inputs. We considered sequences generated from UML state diagrams which models the functional behavior of a system. Model-based mutation analysis was used to evaluate the fault detection potential of the generated sequences.
References
Acree Jr, A. T. (1980). On Mutation. PhD thesis, Georgia Institute of Technology, Atlanta School of Information and Computer Science.
Aichernig, B., Brandl, H., Jobstl, E., Krenn, W., Schlick, R., and Tiran, S. (2015). Momut:: Uml model-based mutation testing for uml. In Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference on, pages 1–8. IEEE.
Ambrosio, A. M., Mattiello-Francisco, F., Santiago Jr, V. A., Silva, W. P., and Martins, E. (2007). Designing fault injection experiments using state-based model to test a space software. In Dependable computing, pages 170–178. Springer.
Cardoso, W. F. F. (2015). Statemutest: uma ferramenta de apoio ao teste baseado em modelos de estado estendidos. Master’s thesis, Universidade Estadual de Campinas.
Cotroneo, D., Di Leo, D., Fucci, F., and Natella, R. (2013). Sabrine: State-based robustness testing of operating systems. In Automated Software Engineering (ASE), 2013 IEEE/ACM 28th International Conference on, pages 125–135. IEEE.
DeMillo, R. and Offutt, A. J. (1991). Constraint-based automatic test data generation. Software Engineering, IEEE Transactions on, 17(9):900–910.
Fabbri, S. C., Delamaro, M. E., Maldonado, J. C., and Masiero, P. C. (1994). Mutation analysis testing for finite state machines. In Software Reliability Engineering, 1994. Proceedings., 5th International Symposium on, pages 220–229. IEEE.
Fabbri, S. C., Maldonado, J. C., and Delamaro, M. E. (1999a). Proteum/fsm: a tool to support finite state machine validation based on mutation testing. In Computer Science Society, 1999. Proceedings. SCCC’99. XIX International Conference of the Chilean, pages 96–104. IEEE.
Fabbri, S. C. P. F., Maldonado, J. C., Sugeta, T., and Masiero, P. C. (1999b). Mutation testing applied to validate specifications based on statecharts. In Software Reliability Engineering, 1999. Proceedings. 10th International Symposium on, pages 210–219. IEEE.
Harman, M., Jia, Y., and Langdon, W. B. (2011). Strong higher order mutation-based test data generation. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 212–222. ACM.
Jia, Y. and Harman, M. (2011). An analysis and survey of the development of mutation testing. Software Engineering, IEEE Transactions on, 37(5):649–678.
Kalaji, A., Hierons, R. M., and Swift, S. (2009). Generating feasible transition paths for testing from an extended finite state machine (efsm). In Software Testing Verification and Validation, 2009. ICST’09. International Conference on, pages 230–239. IEEE.
Kim, Y. G., Hong, H. S., Bae, D.-H., and Cha, S. D. (1999). Test cases generation from uml state diagrams. In Software, IEE Proceedings-, volume 146, pages 187–192. IET.
Martins, E., Sabião, S. B., and Ambrosio, A. M. (1999). Condata: a tool for automating specification-based test case generation for communication systems. Software Quality Journal, 8(4):303–320.
Micskei, Z., Madeira, H., Avritzer, A., Majzik, I., Vieira, M., and Antunes, N. (2012). Robustness testing techniques and tools. In Resilience Assessment and Evaluation of Computing Systems, pages 323–339. Springer.
Offutt, A. J., Rothermel, G., and Zapf, C. (1993). An experimental evaluation of selective mutation. In Proceedings of the 15th international conference on Software Engineering, pages 100–107. IEEE Computer Society Press.
Papadakis, M. and Malevris, N. (2010). Automatic mutation test case generation via dynamic symbolic execution. In Software reliability engineering (ISSRE), 2010 IEEE 21st international symposium on, pages 121–130. IEEE.
Prenninger, W. and Pretschner, A. (2005). Abstractions for model-based testing. Electronic Notes in Theoretical Computer Science, 116:59–71.
Rollet, A. and Salva, S. (2009). Testing robustness of communicating systems using ioco-based approach. In Computers and Communications, 2009. ISCC 2009. IEEE Symposium on, pages 67–72. IEEE.
Rumbaugh, J., Jacobson, I., and Booch, G. (2004). Unified Modeling Language Reference Manual, The. Pearson Higher Education.
Saad-Khorchef, F., Rollet, A., and Castanet, R. (2007). A framework and a tool for robustness testing of communicating software. In Proceedings of the 2007 ACM symposium on Applied computing, pages 1461–1466. ACM.
Samuel, P., Mall, R., and Bothra, A. K. (2008). Automatic test case generation using unified modeling language (uml) state diagrams. Software, IET, 2(2):79–93.
Santiago, V., Amaral, A. S. M. d., Vijaykumar, N. L., Mattiello-Francisco, M. F., Martins, E., and Lopes, O. C. (2006). A practical approach for automated test case generation using statecharts. In Computer Software and Applications Conference, 2006. COMPSAC’06. 30th Annual International, volume 2, pages 183–188. IEEE.
Shirole, M., Suthar, A., and Kumar, R. (2011). Generation of improved test cases from uml state diagram using genetic algorithm. In Proceedings of the 4th India Software Engineering Conference, pages 125–134. ACM.
Sugeta, T. (1999). PROTEUM-RS/ST: uma ferramenta para apoiar a validação de especificações statecharts baseada na análise de mutantes. PhD thesis, Universidade de São Paulo.
Walpole, R. E., Myers, R. H., Myers, S. L., and Ye, K. (2011). Probability and statistics for engineers and scientists, volume 9. Prentice Hall.
Yano, T., Martins, E., and De Sousa, F. L. (2010). Generating feasible test paths from an executable model using a multi-objective approach. In Software Testing, Verification, and Validation Workshops (ICSTW), 2010 Third International Conference on, pages 236–239. IEEE.
Yano, T., Martins, E., and De Sousa, F. L. (2011a). A model-based approach for robustness test generation. In Dependable Computing Workshops (LADCW), 2011 Fifth Latin-American Symposium on, pages 33–34. IEEE.
Yano, T., Martins, E., and De Sousa, F. L. (2011b). Most: a multi-objective search-based testing from efsm. In Software Testing, Verification and Validation Workshops (ICSTW), 2011 IEEE Fourth International Conference on, pages 164–173. IEEE.
Zhang, J., Yang, R., Chen, Z., Zhao, Z., and Xu, B. (2012). Automated efsm-based test case generation with scatter search. In Proceedings of the 7th International Workshop on Automation of Software Test, pages 76–82. IEEE Press.
Aichernig, B., Brandl, H., Jobstl, E., Krenn, W., Schlick, R., and Tiran, S. (2015). Momut:: Uml model-based mutation testing for uml. In Software Testing, Verification and Validation (ICST), 2015 IEEE 8th International Conference on, pages 1–8. IEEE.
Ambrosio, A. M., Mattiello-Francisco, F., Santiago Jr, V. A., Silva, W. P., and Martins, E. (2007). Designing fault injection experiments using state-based model to test a space software. In Dependable computing, pages 170–178. Springer.
Cardoso, W. F. F. (2015). Statemutest: uma ferramenta de apoio ao teste baseado em modelos de estado estendidos. Master’s thesis, Universidade Estadual de Campinas.
Cotroneo, D., Di Leo, D., Fucci, F., and Natella, R. (2013). Sabrine: State-based robustness testing of operating systems. In Automated Software Engineering (ASE), 2013 IEEE/ACM 28th International Conference on, pages 125–135. IEEE.
DeMillo, R. and Offutt, A. J. (1991). Constraint-based automatic test data generation. Software Engineering, IEEE Transactions on, 17(9):900–910.
Fabbri, S. C., Delamaro, M. E., Maldonado, J. C., and Masiero, P. C. (1994). Mutation analysis testing for finite state machines. In Software Reliability Engineering, 1994. Proceedings., 5th International Symposium on, pages 220–229. IEEE.
Fabbri, S. C., Maldonado, J. C., and Delamaro, M. E. (1999a). Proteum/fsm: a tool to support finite state machine validation based on mutation testing. In Computer Science Society, 1999. Proceedings. SCCC’99. XIX International Conference of the Chilean, pages 96–104. IEEE.
Fabbri, S. C. P. F., Maldonado, J. C., Sugeta, T., and Masiero, P. C. (1999b). Mutation testing applied to validate specifications based on statecharts. In Software Reliability Engineering, 1999. Proceedings. 10th International Symposium on, pages 210–219. IEEE.
Harman, M., Jia, Y., and Langdon, W. B. (2011). Strong higher order mutation-based test data generation. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 212–222. ACM.
Jia, Y. and Harman, M. (2011). An analysis and survey of the development of mutation testing. Software Engineering, IEEE Transactions on, 37(5):649–678.
Kalaji, A., Hierons, R. M., and Swift, S. (2009). Generating feasible transition paths for testing from an extended finite state machine (efsm). In Software Testing Verification and Validation, 2009. ICST’09. International Conference on, pages 230–239. IEEE.
Kim, Y. G., Hong, H. S., Bae, D.-H., and Cha, S. D. (1999). Test cases generation from uml state diagrams. In Software, IEE Proceedings-, volume 146, pages 187–192. IET.
Martins, E., Sabião, S. B., and Ambrosio, A. M. (1999). Condata: a tool for automating specification-based test case generation for communication systems. Software Quality Journal, 8(4):303–320.
Micskei, Z., Madeira, H., Avritzer, A., Majzik, I., Vieira, M., and Antunes, N. (2012). Robustness testing techniques and tools. In Resilience Assessment and Evaluation of Computing Systems, pages 323–339. Springer.
Offutt, A. J., Rothermel, G., and Zapf, C. (1993). An experimental evaluation of selective mutation. In Proceedings of the 15th international conference on Software Engineering, pages 100–107. IEEE Computer Society Press.
Papadakis, M. and Malevris, N. (2010). Automatic mutation test case generation via dynamic symbolic execution. In Software reliability engineering (ISSRE), 2010 IEEE 21st international symposium on, pages 121–130. IEEE.
Prenninger, W. and Pretschner, A. (2005). Abstractions for model-based testing. Electronic Notes in Theoretical Computer Science, 116:59–71.
Rollet, A. and Salva, S. (2009). Testing robustness of communicating systems using ioco-based approach. In Computers and Communications, 2009. ISCC 2009. IEEE Symposium on, pages 67–72. IEEE.
Rumbaugh, J., Jacobson, I., and Booch, G. (2004). Unified Modeling Language Reference Manual, The. Pearson Higher Education.
Saad-Khorchef, F., Rollet, A., and Castanet, R. (2007). A framework and a tool for robustness testing of communicating software. In Proceedings of the 2007 ACM symposium on Applied computing, pages 1461–1466. ACM.
Samuel, P., Mall, R., and Bothra, A. K. (2008). Automatic test case generation using unified modeling language (uml) state diagrams. Software, IET, 2(2):79–93.
Santiago, V., Amaral, A. S. M. d., Vijaykumar, N. L., Mattiello-Francisco, M. F., Martins, E., and Lopes, O. C. (2006). A practical approach for automated test case generation using statecharts. In Computer Software and Applications Conference, 2006. COMPSAC’06. 30th Annual International, volume 2, pages 183–188. IEEE.
Shirole, M., Suthar, A., and Kumar, R. (2011). Generation of improved test cases from uml state diagram using genetic algorithm. In Proceedings of the 4th India Software Engineering Conference, pages 125–134. ACM.
Sugeta, T. (1999). PROTEUM-RS/ST: uma ferramenta para apoiar a validação de especificações statecharts baseada na análise de mutantes. PhD thesis, Universidade de São Paulo.
Walpole, R. E., Myers, R. H., Myers, S. L., and Ye, K. (2011). Probability and statistics for engineers and scientists, volume 9. Prentice Hall.
Yano, T., Martins, E., and De Sousa, F. L. (2010). Generating feasible test paths from an executable model using a multi-objective approach. In Software Testing, Verification, and Validation Workshops (ICSTW), 2010 Third International Conference on, pages 236–239. IEEE.
Yano, T., Martins, E., and De Sousa, F. L. (2011a). A model-based approach for robustness test generation. In Dependable Computing Workshops (LADCW), 2011 Fifth Latin-American Symposium on, pages 33–34. IEEE.
Yano, T., Martins, E., and De Sousa, F. L. (2011b). Most: a multi-objective search-based testing from efsm. In Software Testing, Verification and Validation Workshops (ICSTW), 2011 IEEE Fourth International Conference on, pages 164–173. IEEE.
Zhang, J., Yang, R., Chen, Z., Zhao, Z., and Xu, B. (2012). Automated efsm-based test case generation with scatter search. In Proceedings of the 7th International Workshop on Automation of Software Test, pages 76–82. IEEE Press.
Published
2016-05-30
How to Cite
CARDOSO, Wallace Felipe Francisco; MARTINS, Eliane.
Uma avaliação do potencial de detecção de defeitos dos casos de teste de robustez de acordo com a análise de mutantes. In: FAULT TOLERANCE WORKSHOP (WTF), 17. , 2016, Salvador/BA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2016
.
p. 35-48.
ISSN 2595-2684.
DOI: https://doi.org/10.5753/wtf.2016.22874.
