Enhancing widget recognition for automated Android testing using Computer Vision techniques
Resumo
Widget recognition is crucial for automated Android black box testing. Over the past ten years, different industrial tools and academic works have been available for identifying Graphical User Interface (GUI) components in Android screens. Traditional identification methods, like GUI hierarchy parsing, often struggle with dynamic content and complex structures. In contrast, Computer Vision (CV) techniques provide greater robustness and flexibility to adapt to different screen resolutions, design specifications, and patterns. However, the CV-based solutions available are still limited concerning the variety of widgets that can be recognized. Moreover, the current identification of GUI components mainly relies on classification, which can lead to ambiguous lists with repeated elements. In this paper, we combine different CV-based techniques to extract context-based descriptions for each widget, to enhance the identification process by going beyond class recognition for describing widgets. We also implemented two primary CV-based approaches for widget recognition: Object Detection combined with Classification, and a One-Stage Recognition method. We trained and evaluated the approaches on a custom 105 classes widget dataset. Moreover, we present a Computer Vision-based method for describing widgets using their contextual meaning on Android screen captures.
Palavras-chave:
Automated Android Testing, Test Portability, Object Recognition, Object Classification, Optical Character Recognition, Graphical User Interface component, Widget Recognition
Referências
A. A. Abdelhamid, S. Alotaibi, and A. Mousa. 2020. Deep learning-based prototyping of android GUI from hand-drawn mockups. IET Software 14 (2020), 816–824. Issue 7. DOI: 10.1049/iet-sen.2019.0378
David Adamo, Md Khorrom Khan, Sreedevi Koppula, and Renée Bryce. 2018. Reinforcement learning for Android GUI testing. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation (Lake Buena Vista, FL, USA) (A-TEST 2018). Association for Computing Machinery, New York, NY, USA, 2–8. DOI: 10.1145/3278186.3278187
Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M. Memon. 2012. Using GUI ripping for automated testing of Android applications. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering (Essen, Germany) (ASE’12). Association for Computing Machinery, New York, NY, USA, 258–261. DOI: 10.1145/2351676.2351717
Domenico Amalfitano, Vincenzo Riccio, Nicola Amatucci, Vincenzo De Simone, and Anna Rita Fasolino. 2019. Combining Automated GUI Exploration of Android apps with Capture and Replay through Machine Learning. Information and Software Technology 105 (2019), 95–116. DOI: 10.1016/j.infsof.2018.08.007
Luca Ardito, Riccardo Coppola, Simone Leonardi, Maurizio Morisio, and Ugo Buy. 2020. Automated Test Selection for Android Apps Based on APK and Activity Classification. IEEE Access 8 (2020), 187648–187670. DOI: 10.1109/ACCESS.2020.3029735
Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of android apps. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications (Indianapolis, Indiana, USA) (OOPSLA ’13). Association for Computing Machinery, New York, NY, USA, 641–660. DOI: 10.1145/2509136.2509549
Young-Min Baek and Doo-Hwan Bae. 2016. Automated model-based Android GUI testing using multi-level GUI comparison criteria. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore) (ASE ’16). Association for Computing Machinery, New York, NY, USA, 238–249. DOI: 10.1145/2970276.2970313
Sara Banian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy El-Nasr. 2021. VINS: Visual Search for Mobile User Interface Design. [link]
Sara Banian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy El-Nasr. 2021. VINS: Visual Search for Mobile User Interface Design. 1–14. DOI: 10.1145/3411764.3445762
Manuele Barraco, Matteo Stefanini, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. 2022. CaMEL: Mean Teacher Learning for Image Captioning. arXiv:2202.10492
Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020. Unblind your apps: predicting natural-language labels for mobile GUI components by deep learning. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. ACM. DOI: 10.1145/3377811.3380327
Jing Cheng, Dingmei Tan, Tao Zhang, Aodi Wei, Jingyi Chen, and Ali Khattak Hasan. 2022. YOLOv5-MGC: GUI Element Identification for Mobile Applications Based on Improved YOLOv5. Mob. Inf. Syst. 2022 (jan 2022), 9 pages. DOI: 10.1155/2022/8900734
Eliane Collins, Arilo Neto, Auri Vincenzi, and José Maldonado. 2021. Deep Reinforcement Learning based Android Application GUI Testing. In Proceedings of the XXXV Brazilian Symposium on Software Engineering (Joinville, Brazil) (SBES ’21). Association for Computing Machinery, New York, NY, USA, 186–194. DOI: 10.1145/3474624.3474634
Richard Hada Degaki, Juan Gabriel Colonna, Yadini Lopez, José Reginaldo Carvalho, and Edson Silva. 2022. Real Time Detection of Mobile Graphical User Interface Elements Using Convolutional Neural Networks. In Proceedings of the Brazilian Symposium on Multimedia and the Web (Curitiba, Brazil) (WebMedia ’22). Association for Computing Machinery, New York, NY, USA, 159–167. DOI: 10.1145/3539637.3558044
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 845–854. DOI: 10.1145/3126594.3126651
Giovanni Denaro, Luca Guglielmo, Leonardo Mariani, and Oliviero Riganelli. 2019. GUI testing in production: challenges and opportunities. In Companion Proceedings of the 3rd International Conference on the Art, Science, and Engineering of Programming (Genova, Italy) (Programming ’19). Association for Computing Machinery, New York, NY, USA, Article 18, 3 pages. DOI: 10.1145/33.284.333.328.452
Android Developers. 2024. Espresso. [link]
Android Developers. 2024. UI Automator. [link]
Juha Eskonen, Julen Kahles, and Joel Reijonen. 2020. Automating GUI Testing with Image-Based Deep Reinforcement Learning. In 2020 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS). 160–167. DOI: 10.1109/ACSOS49614.2020.00038
Anna Esparcia-Alcazar, F. (Francisco) Almenar, Mirella Martınez, U. (Urko) Rueda, and T.E.J. Vos. 2016. Q-learning strategies for action selection in the TESTAR automated testing tool. In Proceedings of the 6TH International Conference on Metaheuristics and Nature Inspired Computing. 174–180. 6th International Conference on Metaheuristic and Nature inspired Computing, META 2016; [link]
OpenJS Foundation. 2012. Appium. [link]
Jerry Gao, ShiTing Li, Chuanqi Tao, Yejun He, Amrutha Pavani Anumalasetty, Erica Wilson Joseph, Akshata Hatwar Kumbashi Sripathi, and Himabindu Nayani. 2022. An Approach to GUI Test Scenario Generation Using Machine Learning. In 2022 IEEE International Conference On Artificial Intelligence Testing (AITest). 79–86. DOI: 10.1109/AITest55621.2022.00020
Xin Gao, Sundaresh Ram, and Jeffrey J. Rodríguez. 2020. Object sieving and morphological closing to reduce false detections in wide-area aerial imagery. CoRR abs/2010.15260 (2020). arXiv:2010.15260 [link]
Data Driven Design Group. 2024. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. [link]
Shuai Hao, Bin Liu, Suman Nath, William G.J. Halfond, and Ramesh Govindan. 2014. PUMA: programmable UI-automation for large-scale dynamic analysis of mobile apps. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (Bretton Woods, New Hampshire, USA) (MobiSys ’14). Association for Computing Machinery, New York, NY, USA, 204–217. DOI: 10.1145/2594368.2594390
L. V. Haoyin. 2017. Automatic android application GUI testing—-A random walk approach. In 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET). 72–76. DOI: 10.1109/WiSPNET.2017.8299722
Saad Hassan, Manan Arya, Ujjwal Bhardwaj, and Silica Kole. 2018. Extraction and classification of user interface components from an image. International Journal of Pure and Applied Mathematics 118, 24 (2018), 1–16.
Gang Hu, Linjie Zhu, and Junfeng Yang. 2018. AppFlow: using machine learning to synthesize robust, reusable UI tests. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Lake Buena Vista, FL, USA) (ESEC/FSE 2018). Association for Computing Machinery, New York, NY, USA, 269–282. DOI: 10.1145/3236024.3236055
Jianjun Huang, Zhichun Li, Xusheng Xiao, Zhenyu Wu, Kangjie Lu, Xiangyu Zhang, and Guofei Jiang. 2015. SUPOR: Precise and scalable sensitive user input detection for android apps. In Proceedings of the 24th USENIX Conference on Security Symposium (Washington, D.C.) (SEC’15). USENIX Association, USA, 977–992.
IBM. 2021. Rational Functional Tester. [link]
K. Jaganeshwari and S. Djodilatchoumy. 2021. A Novel approach of GUI Mapping with image based widget detection and classification. In 2021 2nd International Conference on Intelligent Engineering and Management (ICIEM). 342–346. DOI: 10.1109/ICIEM51511.2021.9445281
Jinseong Jeon and Jeffrey Foster. 2012. Troyd: Integration Testing for Android. Technical Report CS-TR-5013, Department of Computer Science, University of Maryland, College Park (2012).
Yavuz Koroglu, Alper Sen, Ozlem Muslu, Yunus Mete, Ceyda Ulker, Tolga Tanriverdi, and Yunus Donmez. 2018. QBE: QLearning-Based Exploration of Android Applications. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). 105–115. DOI: 10.1109/ICST.2018.00020
Wing Lam, Zhengkai Wu, Dengfeng Li, Wenyu Wang, Haibing Zheng, Hui Luo, Peng Yan, Yuetang Deng, and Tao Xie. 2017. Record and replay for Android: are we there yet in industrial cases?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 854–859. DOI: 10.1145/3106237.3117769
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020. Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements. arXiv:2010.04295
Mario Linares-Vásquez, Kevin Moran, and Denys Poshyvanyk. 2017. Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testing. 399–410. DOI: 10.1109/ICSME.2017.27
Yadini Pérez López, Laís Dib Albuquerque, Gilmar Jóia de F. Costa Júnior, Daniel Lopes Xavier, Juan David Ochoa, and Denizard Dimitri Camargo. 2023. Adapting RICO Dataset for Boosting Graphical User Interface Component Classification for Automated Android Testing. In 2023 10th International Conference on Soft Computing Machine Intelligence (ISCMI). 118–123. DOI: 10.1109/ISCMI59957.2023.10458576
Yadini Pérez López, Juan G. Colonna, Edson De Araujo Silva, Richard Hada Degaki, and Javier Martinez Silva. 2022. Q-funcT: A Reinforcement Learning Approach for Automated Black Box Functionality Testing. In 2022 IEEE 2nd International Conference on Software Engineering and Artificial Intelligence (SEAI). 119–123. DOI: 10.1109/SEAI55746.2022.9832177
Ying Ma, ChuYi Yu, and Ming Yan. 2022. Icon Label Generation for Mobile Applications by Mean Teacher Learning. DOI: 10.21203/rs.3.rs-1888657/v1
Leonardo Mariani, Mauro Pezzè, Oliviero Riganelli, and Mauro Santoro. 2011. AutoBlackTest: a tool for automatic black-box testing. In 2011 33rd International Conference on Software Engineering (ICSE). 1013–1015. DOI: 10.1145/1985793.1985979
Forough Mehralian, Navid Salehnamadi, and Sam Malek. 2021. Data-driven accessibility repair revisited: on the effectiveness of generating labels for icons in Android apps. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Athens, Greece) (ESEC/FSE 2021). Association for Computing Machinery, New York, NY, USA, 107–118. DOI: 10.1145/3468264.3468604
Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018. The ReDraw Dataset: A Set of Android Screenshots, GUI Metadata, and Labeled Images of GUI Components. [link]
Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2020. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. IEEE Transactions on Software Engineering 46, 2 (2020), 196–221. DOI: 10.1109/TSE.2018.2844788
Seong-Guk Nam and Yeong-Seok Seo. 2023. GUI Component Detection-Based Automated Software Crash Diagnosis. Electronics 12, 11 (2023). DOI: 10.3390/electronics12112382
Clemens Neudecker, Konstantin Baierer, Mike Gerber, Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. 2021. A survey of OCR evaluation tools and metrics. In Proceedings of the 6th International Workshop on Historical Document Imaging and Processing (Lausanne, Switzerland) (HIP ’21). Association for Computing Machinery, New York, NY, USA, 13–18. DOI: 10.1145/3476887.3476888
Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI (T). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). 248–259. DOI: 10.1109/ASE.2015.32
Rafael Padilla, Sergio L. Netto, and Eduardo A. B. da Silva. 2020. A Survey on Performance Metrics for Object-Detection Algorithms. In 2020 International Conference on Systems, Signals and Image Processing (IWSSIP). 237–242. DOI: 10.1109/IWSSIP48289.2020.9145130
Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of Android applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual Event, USA) (ISSTA 2020). Association for Computing Machinery, New York, NY, USA, 153–164. DOI: 10.1145/3395363.3397354
Mahmood R, Mirzaei N, and Malek S. 2014. Automated Test Generation for Graphical User Interfaces: Challenges and Opportunities. ACM Computing Surveys (CSUR) (2014), 45.
Dillon Reis, Jordan Kupec, Jacqueline Hong, and Ahmad Daoudi. 2024. Real-Time Flying Object Detection with YOLOv8. (2024). DOI: 10.48550/arXiv.2305.09972
Ariel Rosenfeld, Odaya Kardashov, and Orel Zang. 2018. Automation of Android Applications Functional Testing Using Machine Learning Activities Classification. In 2018 IEEE/ACM 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft). 122–132.
Xiaolei Sun, Tongyu Li, and Jianfeng Xu. 2020. UI Components Recognition System Based On Image Understanding. In 2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C). 65–71. DOI: 10.1109/QRS-C51114.2020.00022
Amrita Vazirani. 2022. What is Android UI Testing? [link]
Tuyet Vuong and Shingo Takada. 2019. Semantic analysis for deep Q-network in android GUI testing. In Knowledge Systems Institute Graduate School, 123–128. 31st International Conference on Software Engineering and Knowledge Engineering, SEKE 2019 ; Conference date: 10-07-2019 Through 12-07-2019.Proceedings - SEKE 2019 (Proceedings of the International Conference on Software Engineering and Knowledge Engineering, SEKE). DOI: 10.18293/SEKE2019-080
Thi Anh Tuyet Vuong and Shingo Takada. 2018. A reinforcement learning based approach to automated testing of Android applications. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation (Lake Buena Vista, FL, USA) (A-TEST 2018). Association for Computing Machinery, New York, NY, USA, 31–37. DOI: 10.1145/3278186.3278191
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. arXiv:2108.03353
Thomas D. White, Gordon Fraser, and Guy J. Brown. 2019. Improving random GUI testing with image-based widget detection. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (Beijing, China) (ISSTA 2019). Association for Computing Machinery, New York, NY, USA, 307–317. DOI: 10.1145/3293882.3330551
Xusheng Xiao, Xiaoyin Wang, Zhihao Cao, Hanlin Wang, and Peng Gao. 2019. IconIntent: Automatic Identification of Sensitive UI Widgets Based on Icon Classification for Android Apps. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 257–268. DOI: 10.1109/ICSE.2019.00041
Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020. UIED: a hybrid tool for GUI element detection. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1655–1659. DOI: 10.1145/3368089.3417940
Feng Xue. 2020. Automated mobile apps testing from visual perspective. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual Event, USA) (ISSTA 2020). Association for Computing Machinery, New York, NY, USA, 577–581. DOI: 10.1145/3395363.3402644
Shengcheng Yu, Chunrong Fang, Tongyu Li, Mingzhe Du, Xuan Li, Jing Zhang, Yexiao Yun, Xu Wang, and Zhenyu Chen. 2021. Automated Mobile App Test Script Intent Generation via Image and Code Understanding. arXiv:2107.05165 [cs.SE]
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P. Bigham. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. arXiv:2101.04893
David Adamo, Md Khorrom Khan, Sreedevi Koppula, and Renée Bryce. 2018. Reinforcement learning for Android GUI testing. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation (Lake Buena Vista, FL, USA) (A-TEST 2018). Association for Computing Machinery, New York, NY, USA, 2–8. DOI: 10.1145/3278186.3278187
Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M. Memon. 2012. Using GUI ripping for automated testing of Android applications. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering (Essen, Germany) (ASE’12). Association for Computing Machinery, New York, NY, USA, 258–261. DOI: 10.1145/2351676.2351717
Domenico Amalfitano, Vincenzo Riccio, Nicola Amatucci, Vincenzo De Simone, and Anna Rita Fasolino. 2019. Combining Automated GUI Exploration of Android apps with Capture and Replay through Machine Learning. Information and Software Technology 105 (2019), 95–116. DOI: 10.1016/j.infsof.2018.08.007
Luca Ardito, Riccardo Coppola, Simone Leonardi, Maurizio Morisio, and Ugo Buy. 2020. Automated Test Selection for Android Apps Based on APK and Activity Classification. IEEE Access 8 (2020), 187648–187670. DOI: 10.1109/ACCESS.2020.3029735
Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and depth-first exploration for systematic testing of android apps. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications (Indianapolis, Indiana, USA) (OOPSLA ’13). Association for Computing Machinery, New York, NY, USA, 641–660. DOI: 10.1145/2509136.2509549
Young-Min Baek and Doo-Hwan Bae. 2016. Automated model-based Android GUI testing using multi-level GUI comparison criteria. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore) (ASE ’16). Association for Computing Machinery, New York, NY, USA, 238–249. DOI: 10.1145/2970276.2970313
Sara Banian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy El-Nasr. 2021. VINS: Visual Search for Mobile User Interface Design. [link]
Sara Banian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy El-Nasr. 2021. VINS: Visual Search for Mobile User Interface Design. 1–14. DOI: 10.1145/3411764.3445762
Manuele Barraco, Matteo Stefanini, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. 2022. CaMEL: Mean Teacher Learning for Image Captioning. arXiv:2202.10492
Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020. Unblind your apps: predicting natural-language labels for mobile GUI components by deep learning. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. ACM. DOI: 10.1145/3377811.3380327
Jing Cheng, Dingmei Tan, Tao Zhang, Aodi Wei, Jingyi Chen, and Ali Khattak Hasan. 2022. YOLOv5-MGC: GUI Element Identification for Mobile Applications Based on Improved YOLOv5. Mob. Inf. Syst. 2022 (jan 2022), 9 pages. DOI: 10.1155/2022/8900734
Eliane Collins, Arilo Neto, Auri Vincenzi, and José Maldonado. 2021. Deep Reinforcement Learning based Android Application GUI Testing. In Proceedings of the XXXV Brazilian Symposium on Software Engineering (Joinville, Brazil) (SBES ’21). Association for Computing Machinery, New York, NY, USA, 186–194. DOI: 10.1145/3474624.3474634
Richard Hada Degaki, Juan Gabriel Colonna, Yadini Lopez, José Reginaldo Carvalho, and Edson Silva. 2022. Real Time Detection of Mobile Graphical User Interface Elements Using Convolutional Neural Networks. In Proceedings of the Brazilian Symposium on Multimedia and the Web (Curitiba, Brazil) (WebMedia ’22). Association for Computing Machinery, New York, NY, USA, 159–167. DOI: 10.1145/3539637.3558044
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 845–854. DOI: 10.1145/3126594.3126651
Giovanni Denaro, Luca Guglielmo, Leonardo Mariani, and Oliviero Riganelli. 2019. GUI testing in production: challenges and opportunities. In Companion Proceedings of the 3rd International Conference on the Art, Science, and Engineering of Programming (Genova, Italy) (Programming ’19). Association for Computing Machinery, New York, NY, USA, Article 18, 3 pages. DOI: 10.1145/33.284.333.328.452
Android Developers. 2024. Espresso. [link]
Android Developers. 2024. UI Automator. [link]
Juha Eskonen, Julen Kahles, and Joel Reijonen. 2020. Automating GUI Testing with Image-Based Deep Reinforcement Learning. In 2020 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS). 160–167. DOI: 10.1109/ACSOS49614.2020.00038
Anna Esparcia-Alcazar, F. (Francisco) Almenar, Mirella Martınez, U. (Urko) Rueda, and T.E.J. Vos. 2016. Q-learning strategies for action selection in the TESTAR automated testing tool. In Proceedings of the 6TH International Conference on Metaheuristics and Nature Inspired Computing. 174–180. 6th International Conference on Metaheuristic and Nature inspired Computing, META 2016; [link]
OpenJS Foundation. 2012. Appium. [link]
Jerry Gao, ShiTing Li, Chuanqi Tao, Yejun He, Amrutha Pavani Anumalasetty, Erica Wilson Joseph, Akshata Hatwar Kumbashi Sripathi, and Himabindu Nayani. 2022. An Approach to GUI Test Scenario Generation Using Machine Learning. In 2022 IEEE International Conference On Artificial Intelligence Testing (AITest). 79–86. DOI: 10.1109/AITest55621.2022.00020
Xin Gao, Sundaresh Ram, and Jeffrey J. Rodríguez. 2020. Object sieving and morphological closing to reduce false detections in wide-area aerial imagery. CoRR abs/2010.15260 (2020). arXiv:2010.15260 [link]
Data Driven Design Group. 2024. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. [link]
Shuai Hao, Bin Liu, Suman Nath, William G.J. Halfond, and Ramesh Govindan. 2014. PUMA: programmable UI-automation for large-scale dynamic analysis of mobile apps. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (Bretton Woods, New Hampshire, USA) (MobiSys ’14). Association for Computing Machinery, New York, NY, USA, 204–217. DOI: 10.1145/2594368.2594390
L. V. Haoyin. 2017. Automatic android application GUI testing—-A random walk approach. In 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET). 72–76. DOI: 10.1109/WiSPNET.2017.8299722
Saad Hassan, Manan Arya, Ujjwal Bhardwaj, and Silica Kole. 2018. Extraction and classification of user interface components from an image. International Journal of Pure and Applied Mathematics 118, 24 (2018), 1–16.
Gang Hu, Linjie Zhu, and Junfeng Yang. 2018. AppFlow: using machine learning to synthesize robust, reusable UI tests. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Lake Buena Vista, FL, USA) (ESEC/FSE 2018). Association for Computing Machinery, New York, NY, USA, 269–282. DOI: 10.1145/3236024.3236055
Jianjun Huang, Zhichun Li, Xusheng Xiao, Zhenyu Wu, Kangjie Lu, Xiangyu Zhang, and Guofei Jiang. 2015. SUPOR: Precise and scalable sensitive user input detection for android apps. In Proceedings of the 24th USENIX Conference on Security Symposium (Washington, D.C.) (SEC’15). USENIX Association, USA, 977–992.
IBM. 2021. Rational Functional Tester. [link]
K. Jaganeshwari and S. Djodilatchoumy. 2021. A Novel approach of GUI Mapping with image based widget detection and classification. In 2021 2nd International Conference on Intelligent Engineering and Management (ICIEM). 342–346. DOI: 10.1109/ICIEM51511.2021.9445281
Jinseong Jeon and Jeffrey Foster. 2012. Troyd: Integration Testing for Android. Technical Report CS-TR-5013, Department of Computer Science, University of Maryland, College Park (2012).
Yavuz Koroglu, Alper Sen, Ozlem Muslu, Yunus Mete, Ceyda Ulker, Tolga Tanriverdi, and Yunus Donmez. 2018. QBE: QLearning-Based Exploration of Android Applications. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). 105–115. DOI: 10.1109/ICST.2018.00020
Wing Lam, Zhengkai Wu, Dengfeng Li, Wenyu Wang, Haibing Zheng, Hui Luo, Peng Yan, Yuetang Deng, and Tao Xie. 2017. Record and replay for Android: are we there yet in industrial cases?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 854–859. DOI: 10.1145/3106237.3117769
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020. Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements. arXiv:2010.04295
Mario Linares-Vásquez, Kevin Moran, and Denys Poshyvanyk. 2017. Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testing. 399–410. DOI: 10.1109/ICSME.2017.27
Yadini Pérez López, Laís Dib Albuquerque, Gilmar Jóia de F. Costa Júnior, Daniel Lopes Xavier, Juan David Ochoa, and Denizard Dimitri Camargo. 2023. Adapting RICO Dataset for Boosting Graphical User Interface Component Classification for Automated Android Testing. In 2023 10th International Conference on Soft Computing Machine Intelligence (ISCMI). 118–123. DOI: 10.1109/ISCMI59957.2023.10458576
Yadini Pérez López, Juan G. Colonna, Edson De Araujo Silva, Richard Hada Degaki, and Javier Martinez Silva. 2022. Q-funcT: A Reinforcement Learning Approach for Automated Black Box Functionality Testing. In 2022 IEEE 2nd International Conference on Software Engineering and Artificial Intelligence (SEAI). 119–123. DOI: 10.1109/SEAI55746.2022.9832177
Ying Ma, ChuYi Yu, and Ming Yan. 2022. Icon Label Generation for Mobile Applications by Mean Teacher Learning. DOI: 10.21203/rs.3.rs-1888657/v1
Leonardo Mariani, Mauro Pezzè, Oliviero Riganelli, and Mauro Santoro. 2011. AutoBlackTest: a tool for automatic black-box testing. In 2011 33rd International Conference on Software Engineering (ICSE). 1013–1015. DOI: 10.1145/1985793.1985979
Forough Mehralian, Navid Salehnamadi, and Sam Malek. 2021. Data-driven accessibility repair revisited: on the effectiveness of generating labels for icons in Android apps. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Athens, Greece) (ESEC/FSE 2021). Association for Computing Machinery, New York, NY, USA, 107–118. DOI: 10.1145/3468264.3468604
Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018. The ReDraw Dataset: A Set of Android Screenshots, GUI Metadata, and Labeled Images of GUI Components. [link]
Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2020. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. IEEE Transactions on Software Engineering 46, 2 (2020), 196–221. DOI: 10.1109/TSE.2018.2844788
Seong-Guk Nam and Yeong-Seok Seo. 2023. GUI Component Detection-Based Automated Software Crash Diagnosis. Electronics 12, 11 (2023). DOI: 10.3390/electronics12112382
Clemens Neudecker, Konstantin Baierer, Mike Gerber, Christian Clausner, Apostolos Antonacopoulos, and Stefan Pletschacher. 2021. A survey of OCR evaluation tools and metrics. In Proceedings of the 6th International Workshop on Historical Document Imaging and Processing (Lausanne, Switzerland) (HIP ’21). Association for Computing Machinery, New York, NY, USA, 13–18. DOI: 10.1145/3476887.3476888
Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI (T). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). 248–259. DOI: 10.1109/ASE.2015.32
Rafael Padilla, Sergio L. Netto, and Eduardo A. B. da Silva. 2020. A Survey on Performance Metrics for Object-Detection Algorithms. In 2020 International Conference on Systems, Signals and Image Processing (IWSSIP). 237–242. DOI: 10.1109/IWSSIP48289.2020.9145130
Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of Android applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual Event, USA) (ISSTA 2020). Association for Computing Machinery, New York, NY, USA, 153–164. DOI: 10.1145/3395363.3397354
Mahmood R, Mirzaei N, and Malek S. 2014. Automated Test Generation for Graphical User Interfaces: Challenges and Opportunities. ACM Computing Surveys (CSUR) (2014), 45.
Dillon Reis, Jordan Kupec, Jacqueline Hong, and Ahmad Daoudi. 2024. Real-Time Flying Object Detection with YOLOv8. (2024). DOI: 10.48550/arXiv.2305.09972
Ariel Rosenfeld, Odaya Kardashov, and Orel Zang. 2018. Automation of Android Applications Functional Testing Using Machine Learning Activities Classification. In 2018 IEEE/ACM 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft). 122–132.
Xiaolei Sun, Tongyu Li, and Jianfeng Xu. 2020. UI Components Recognition System Based On Image Understanding. In 2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C). 65–71. DOI: 10.1109/QRS-C51114.2020.00022
Amrita Vazirani. 2022. What is Android UI Testing? [link]
Tuyet Vuong and Shingo Takada. 2019. Semantic analysis for deep Q-network in android GUI testing. In Knowledge Systems Institute Graduate School, 123–128. 31st International Conference on Software Engineering and Knowledge Engineering, SEKE 2019 ; Conference date: 10-07-2019 Through 12-07-2019.Proceedings - SEKE 2019 (Proceedings of the International Conference on Software Engineering and Knowledge Engineering, SEKE). DOI: 10.18293/SEKE2019-080
Thi Anh Tuyet Vuong and Shingo Takada. 2018. A reinforcement learning based approach to automated testing of Android applications. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation (Lake Buena Vista, FL, USA) (A-TEST 2018). Association for Computing Machinery, New York, NY, USA, 31–37. DOI: 10.1145/3278186.3278191
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. arXiv:2108.03353
Thomas D. White, Gordon Fraser, and Guy J. Brown. 2019. Improving random GUI testing with image-based widget detection. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (Beijing, China) (ISSTA 2019). Association for Computing Machinery, New York, NY, USA, 307–317. DOI: 10.1145/3293882.3330551
Xusheng Xiao, Xiaoyin Wang, Zhihao Cao, Hanlin Wang, and Peng Gao. 2019. IconIntent: Automatic Identification of Sensitive UI Widgets Based on Icon Classification for Android Apps. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 257–268. DOI: 10.1109/ICSE.2019.00041
Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020. UIED: a hybrid tool for GUI element detection. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1655–1659. DOI: 10.1145/3368089.3417940
Feng Xue. 2020. Automated mobile apps testing from visual perspective. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual Event, USA) (ISSTA 2020). Association for Computing Machinery, New York, NY, USA, 577–581. DOI: 10.1145/3395363.3402644
Shengcheng Yu, Chunrong Fang, Tongyu Li, Mingzhe Du, Xuan Li, Jing Zhang, Yexiao Yun, Xu Wang, and Zhenyu Chen. 2021. Automated Mobile App Test Script Intent Generation via Image and Code Understanding. arXiv:2107.05165 [cs.SE]
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P. Bigham. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. arXiv:2101.04893
Publicado
14/10/2024
Como Citar
LÓPEZ, Yadini Pérez; ALBUQUERQUE, Laís Dib; COSTA JÚNIOR, Gilmar J. F.; XAVIER, Daniel Lopes; BALBI, Leticia; COLONNA, Juan G.; DEGAKI, Richand.
Enhancing widget recognition for automated Android testing using Computer Vision techniques. In: BRAZILIAN SYMPOSIUM ON MULTIMEDIA AND THE WEB (WEBMEDIA), 30. , 2024, Juiz de Fora/MG.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 133-143.
DOI: https://doi.org/10.5753/webmedia.2024.243172.
