Euler Horta Marinho
Eduardo Figueiredo
ABSTRACT
The pressing demand for high-quality mobile applications has a major influence on Software Engineering practices, such as testing and debugging. The variety of mobile platforms is permeated with different resources related to communication capabilities, sensors, and user-controlled options. As a result, applications may exhibit unexpected behaviors and resource interactions can introduce failures that manifest themselves in specific resource combinations. These failures can affect the quality of mobile applications and degrade the user experience. To reduce human effort of manual debugging, several techniques have been proposed and developed aiming to partially or fully automate fault localization. Fault localization techniques, such as Spectrum-based Fault Localization (SBFL), identify suspicious faulty program elements related to a software failure. However, we still lack empirical knowledge about the applicability of fault localization techniques in the context of mobile applications, specifically considering resource interaction failures. To address this problem, this paper evaluates the use of SBFL aiming to locate faults in 8 Android applications and verify the sensitivity of SBFL to variations in resource interactions. We rely on mutation testing to simulate faults and on the Ochiai coefficient as an indicator of the suspicious faulty code. Our results indicate that SBFL is able to rank more than 75% of the faulty code in 6 out of 8 applications. We also observed that the ranking of suspicious code varies depending on the combination of enabled resources (e.g., Wi-Fi and Location) in the mobile applications.
- R. Abreu, P. Zoeteweij, and A. J.C. van Gemund. 2007. On the Accuracy of Spectrum-based Fault Localization. In Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION). 89–98.Google Scholar
- Rui Abreu, P. Zoeteweij, and A. J. C. van Gemund. 2006. An Evaluation of Similarity Coefficients for Software Fault Localization. In Pacific Rim International Symposium on Dependable Computing (PRDC). 39–46.Google Scholar
- AnkiDroid. Accessed 6-May-2023. Anki flashcards on Android. https://github.com/ankidroid/Anki-AndroidGoogle Scholar
- S. Apel, D. Batory, C. Kästner, and G. Saake. 2013. Feature-oriented software product Lines. Springer Berlin / Heidelberg.Google Scholar
- S. Apel, S. Kolesnikov, N. Siegmund, C. Kästner, and B. Garvin. 2013. Exploring Feature Interactions in the Wild: The New Feature-Interaction Challenge. In International Workshop on Feature-Oriented Software Development (FOSD) (Indianapolis, Indiana, USA) (FOSD ’13). 1–8.Google Scholar
- Samuel Benton, Xia Li, Yiling Lou, and Lingming Zhang. 2022. Evaluating and Improving Unified Debugging. IEEE Transactions on Software Engineering 48, 11 (2022), 4692–4716.Google ScholarCross Ref
- T. F. Bowen, FS Dworack, C. Chow, N. Griffeth, G. E Herman, and Y-J Lin. 1989. The feature interaction problem in telecommunications systems. In International Conference on Software Engineering for Telecommunication Switching Systems (SETSS). 59–62.Google Scholar
- M. Ceccato, A. Marchetto, L. Mariani, C. D. Nguyen, and P. Tonella. 2015. Do Automatically Generated Test Cases Make Debugging Easier? An Experimental Assessment of Debugging Effectiveness and Efficiency. ACM Transactions on Software Engineering and Methodology 25, 1, Article 5 (2015), 38 pages.Google ScholarDigital Library
- M. B. Cohen, M. B. Dwyer, and J. Shi. 2007. Interaction Testing of Highly-Configurable Systems in the Presence of Constraints. In ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 129–139.Google Scholar
- J. Dąbrowski, E. Letier, A. Perini, and A. Susi. 2022. Analysing app reviews for Software Engineering: a systematic literature review. Empirical Software Engineering 27, 43 (2022), 1–63.Google Scholar
- H. A. de Souza, D. Mutti, M. L. Chaim, and F. Kon. 2018. Contextualizing spectrum-based fault localization. Information and Software Technology 94 (2018), 245–261.Google ScholarDigital Library
- J. P. Diniz, C. -P. Wong, C. Kästner, and E. Figueiredo. 2021. Dissecting Strongly Subsuming Second-Order Mutants. In IEEE International Conference on Software Testing, Verification, and Validation (ICST). 171–181.Google Scholar
- C. Escobar-Velásquez, M. Linares-Vásquez, G. Bavota, M. Tufano, K. Moran, M. Di Penta, C. Vendome, C. Bernal-Cárdenas, and D. Poshyvanyk. 2020. Enabling Mutant Generation for Open- and Closed-Source Android Apps. IEEE Transactions on Software Engineering (TSE) 48, 1 (2020), 186–208.Google ScholarDigital Library
- F. Ferreira, G. Vale, J. P. Diniz, and E. Figueiredo. 2021. Evaluating T-wise testing strategies in a community-wide dataset of configurable software systems. Journal of Systems and Software (JSS) (2021), 110990.Google Scholar
- J. A. Galindo, H. Turner, D. Benavides, and J. White. 2016. Testing variability-intensive systems using automated analysis: an application to Android. Software Quality Journal (SQJ) 24 (2016), 365–405.Google ScholarDigital Library
- R. Gopinath, C. Jensen, and A. Groce. 2014. Code Coverage for Suite Evaluation by Developers. In IEEE/ACM International Conference on Software Engineering (ICSE). 72–82.Google Scholar
- Ground. Accessed 6-May-2023. Ground mobile data collection app for Android. https://github.com/google/ground-androidGoogle Scholar
- Mary Jean Harrold, Gregg Rothermel, Rui Wu, and Liu Yi. 1998. An Empirical Investigation of Program Spectra. SIGPLAN Notices 33, 7 (1998), 83–90.Google ScholarDigital Library
- S. Heiden, L. Grunske, T. Kehrer, F. Keller, A. van Hoorn, A. Filieri, and D. Lo. 2019. An evaluation of pure spectrum-based fault localization techniques for large-scale software systems. Software: Practice and Experience 49, 8 (2019), 1197–1224.Google ScholarCross Ref
- K. Holl and F. Elberzhager. 2019. Chapter One - Mobile Application Quality Assurance. Advances in Computers, Vol. 112. Elsevier, 1–77.Google Scholar
- J. A. Jones, M. Jean Harrold, and J. Stasko. 2002. Visualization of Test Information to Assist Fault Localization. In IEEE/ACM International Conference on Software Engineering (ICSE). 467–477.Google Scholar
- M. C. Júnior, D. Amalfitano, L. Garcés, A. R. Fasolino, Stevão A. Andrade, and M. Delamaro. 2022. Dynamic Testing Techniques of Non-functional Requirements in Mobile Apps: A Systematic Mapping Study. ACM Computing Surveys (CSUR) 54, 10s (2022), 1–38.Google ScholarDigital Library
- P. Kong, L. Li, J. Gao, T. Riom, Y. Zhao, T. F. Bissyandé, and J. Klein. 2021. ANCHOR: locating Android framework-specific crashing faults. Automated Software Engineering 28, 2 (2021), 10.Google ScholarDigital Library
- T. Laurent, M. Papadakis, M. Kintis, C. Henard, Y. Le Le Traon, and A. Ventresque. 2017. Assessing and Improving the Mutation Testing Practice of PIT. In IEEE International Conference on Software Testing, Verification, and Validation (ICST). 430–435.Google Scholar
- M. Linares-Vásquez, C. Bernal-Cardenas, K. Moran, and D. Poshyvanyk. 2017. How do Developers Test Android Applications?. In IEEE International Conference on Software Maintenance and Evolution (ICSME). 613–622.Google Scholar
- Y. Lu, M. Pan, J. Zhai, T. Zhang, and X. Li. 2019. Preference-wise testing for Android applications. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). 268–278.Google Scholar
- C. Luo, J. Goncalves, E. Velloso, and V. Kostakos. 2020. A survey of context simulation for testing mobile context-aware applications. ACM Computing Surveys (CSUR) 53, 1 (2020), 1–39.Google ScholarDigital Library
- P. Machado, J. Campos, and R. Abreu. 2013. MZoltar: Automatic Debugging of Android Applications. In International Workshop on Software Development Lifecycle for Mobile (DeMobile). 9–16.Google Scholar
- E. H. Marinho, F. Ferreira, J. P. Diniz, and E. Figueiredo. 2023. Evaluating testing strategies for resource related failures in mobile applications. Software Quality Journal (2023), 1–27.Google Scholar
- A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. 1996. An Experimental Determination of Sufficient Mutant Operators. ACM Transactions on Software Engineering and Methodology 5, 2 (1996), 99–118.Google ScholarDigital Library
- J. Oliveira, M. Souza, M. Flauzino, R. Durelli, and E. Figueiredo. 2022. Can Source Code Analysis Indicate Programming Skills? A Survey with Developers. In International Conference on the Quality of Information and Communications Technology (QUATIC). 156–171.Google Scholar
- Openscale. Accessed 6-May-2023. Open-source weight and body metrics tracker, with support for Bluetooth scales. https://github.com/oliexdev/openScaleGoogle Scholar
- Owntracks. Accessed 6-May-2023. Open-source weight and body metrics tracker, with support for Bluetooth scales. https://github.com/owntracks/androidGoogle Scholar
- M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman. 2019. Mutation Testing Advances: An Analysis and Survey. Advances in Computers, Vol. 112. Elsevier, 275–378.Google Scholar
- O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn. 2021. A Survey of Flaky Tests. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1, Article 17 (2021), 74 pages.Google Scholar
- S. Pearson, J. Campos, R. Just, G. Fraser, R. Abreu, M. D. Ernst, D. Pang, and B. Keller. 2017. Evaluating and Improving Fault Localization. In IEEE/ACM International Conference on Software Engineering (ICSE). 609–620.Google Scholar
- PocketHub. Accessed 6-May-2023. PocketHub Android App. https://github.com/pockethub/PocketHubGoogle Scholar
- M. Polo-Usaola and I. Rodríguez-Trujillo. 2021. Analysing the combination of cost reduction techniques in Android mutation testing. Software Testing, Verification and Reliability 31, 7 (2021), e1769.Google ScholarCross Ref
- RadioDroid. Accessed 6-May-2023. Radio Browser App. https://github.com/segler-alex/RadioDroidGoogle Scholar
- Q. I. Sarhan and Á. Beszédes. 2022. A Survey of Challenges in Spectrum-Based Software Fault Localization. IEEE Access 10 (2022), 10618–10639.Google ScholarCross Ref
- H. N. Silva, J. Prado Lima, S. R. Vergilio, and A. T. Endo. 2022. A mapping study on mutation testing for mobile applications. Software Testing, Verification and Reliability 32, 8 (2022), e1801.Google ScholarCross Ref
- T. Su, L. Fan, S. Chen, Y. Liu, L. Xu, G. Pu, and Z. Su. 2020. Why my app crashes? Understanding and benchmarking framework-specific exceptions of Android apps. IEEE Transactions on Software Engineering (TSE) (2020), 1115–1137.Google Scholar
- J. Sun, T. Su, J. Li, Z. Dong, G. Pu, T. Xie, and Z. Su. 2021. Understanding and Finding System Setting-Related Defects in Android Apps. In ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 204–215.Google Scholar
- Threema. Accessed 6-May-2023. Threema App for Android. https://github.com/threema-ch/threema-androidGoogle Scholar
- P. Tramontana, D. Amalfitano, N. Amatucci, and A. R. Fasolino. 2019. Automated functional testing of mobile applications: a systematic mapping study. Software Quality Journal (SQJ) 27, 1 (2019), 149–201.Google ScholarDigital Library
- R. H. Untch, A. J. Offutt, and M. J. Harrold. 1993. Mutation Analysis Using Mutant Schemata. In ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 139–148.Google Scholar
- I. K. Villanes, A. T Endo, and A. C. Dias-Neto. 2022. A multivocal literature mapping on mobile compatibility testing. International Journal of Computer Applications in Technology 69, 2 (2022), 173–192.Google ScholarDigital Library
- Y. Wang, M. V. Mäntylä, Z. Liu, J. Markkula, and P. Raulamo-jurvanen. 2022. Improving test automation maturity: A multivocal literature review. Software Testing, Verification and Reliability 32, 3 (2022), e1804.Google ScholarCross Ref
- H. M. Win, S. H. Tan, and Y. Sui. 2023. Event-aware precise dynamic slicing for automatic debugging of Android applications. Journal of Systems and Software (JSS) (2023), 111606.Google Scholar
- C. Wohlin, P. Runeson, M. Host, M. C. Ohlsson, B. Regnell, and A. Wesslen. 2012. Experimentation in Software Engineering. Springer Berlin / Heidelberg.Google Scholar
- W. E. Wong, V. Debroy, R. Gao, and Y. Li. 2014. The DStar Method for Effective Software Fault Localization. IEEE Transactions on Reliability 63, 1 (2014), 290–308.Google ScholarCross Ref
- W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa. 2016. A Survey on Software Fault Localization. IEEE Transactions on Software Engineering 42, 8 (2016), 707–740.Google ScholarDigital Library
- WordPress. Accessed 6-May-2023. WordPress for Android. https://github.com/wordpress-mobile/WordPress-AndroidGoogle Scholar
- A. Zakari, S. P. Lee, K. A. Alam, and R. Ahmad. 2019. Software fault localisation: a systematic mapping study. IET Software 13, 1 (2019), 60–74.Google ScholarDigital Library
- C. Zamfir and G. Candea. 2010. Execution Synthesis: A Technique for Automated Software Debugging (EuroSys). In European Conference on Computer Systems. 321–334.Google Scholar
- M. Zhang, Y. Li, X. Li, L. Chen, Y. Zhang, L. Zhang, and S. Khurshid. 2021. An Empirical Study of Boosting Spectrum-Based Fault Localization via PageRank. IEEE Transactions on Software Engineering 47, 6 (2021), 1089–1113.Google ScholarCross Ref
- D. Zou, J. Liang, Y. Xiong, M. D. Ernst, and L. Zhang. 2021. An Empirical Study of Fault Localization Families and Their Combinations. IEEE Transactions on Software Engineering 47, 2 (2021), 332–347.Google ScholarDigital Library
Index Terms
- Applying Spectrum-Based Fault Localization to Android Applications
Recommendations
Fault density, fault types, and spectra-based fault localization
This paper presents multiple empirical experiments that investigate the impact of fault quantity and fault type on statistical, coverage-based fault localization techniques and fault-localization interference. Fault-localization interference is a ...
Mitigating the Effect of Coincidental Correctness in Spectrum Based Fault Localization
ICST '12: Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and ValidationCoincidentally correct test cases are those that execute faulty statements but do not cause failures. Such test cases reduce the effectiveness of spectrum-based fault localization techniques, such as Ochiai. These techniques calculate a suspiciousness ...
Factorising the Multiple Fault Localization Problem: Adapting Single-Fault Localizer to Multi-fault Programs
APSEC '12: Proceedings of the 2012 19th Asia-Pacific Software Engineering Conference - Volume 01Software failures are not rare and fault localizations always an important but laborious activity. Since there is no guarantee that no more than one fault exists in a faulty program, the approach to locate all the faults is necessary. Spectrum-based ...
Comments