Nathalia da Cruz Alves
Christiane Gresse von Wangenheim
ABSTRACT
The layout of graphical user interfaces (GUIs) takes into consideration the distribution of buttons, images, texts, and other components that are important factors for a positive experience of interactivity and usability. Therefore, it may be helpful to consult similar layouts during the design of GUIs for inspiration as well as to direct the development of original interfaces. This may be even more important in the context of the design of mobile applications in an educational context or when being developed by end-users with low-code environments, such as App Inventor. Therefore, this article presents an approach for analyzing the similarity of GUI layout using deep learning. First, GUI components and their positions are detected using the YOLO model. Then, the similarity of layouts is analyzed based on the GUI components detected. Results of the evaluation of the developed approach indicated a good accuracy of the detection of GUI components and the analysis of similarity close to human predictions. The results of this research can be used for the automated analysis of the similarity of GUI layouts of mobile applications, which can be used to analyze the design of mobile applications (consistency), search for similar designs, or assess the originality of a GUI design. It can also be applied in an educational context to assess the development of creativity as part of the development of mobile applications in order to teach computing.
- S. Almrayat, R. Yousef e A. Sharieh, 2019. Evaluating the Impact of GUI Similarity between Android Applications to Measure their Functional Similarity. International Journal of Computer Applications, 975, 8887.Google Scholar
- G. Alves, 2020. Detecção de Objetos com YOLO - Uma abordagem moderna. In: Expert Academy. [S.I.], 13 out. 2020. Disponível em: https://iaexpert.academy/2020/10/13/deteccao-de-objetos-com-yolo-umaabordagem-moderna/. Acesso em: 10 mai. 2022.Google Scholar
- N. da C. Alves, C. Gresse von Wangenheim, J. C. R. Hauck, 2020. Teaching Programming to Novices: A Large-scale Analysis of App Inventor Projects. In: Proc. of the 15th Latin-American Conference on Learning Technologies, Loja/Ecuador.Google Scholar
- S. Amershi, et al, 2019. Software engineering for machine learning: A case study. International Conference on Software Engineering: Software Engineering in Practice. Montreal, QC, Canada: IEEE. p. 291--300.Google ScholarDigital Library
- D. d. S. Baulé, C. Gresse von Wangenheim, A. von Wangenheim, J. C. R. Hauck, 2020. Recent Progress in Automated Code Generation from GUI Images using Machine Learning Techniques. Journal of Universal Computer Science, 26(9).Google ScholarCross Ref
- D. d. S. Baulé, C. Gresse von Wangenheim, J. C. R. Hauck, A. von Wangenheim, E. C. Vargas Júnior, 2021. Using Deep Learning to Support the User Interface Design of Mobile Applications with App Inventor. In Anais do XX Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais, online.Google Scholar
- F. Behrang, S. P. Reiss, A. Orso, 2018. GUIfetch. Proc. of the 5th Int. Conference on Mobile Software Engineering and Systems, ACM, EUA.Google Scholar
- S. Bunian, K. Li, C. Jemmali, C. Harteveld, Y. Fu, e M. S. Seif El-Nasr, 2021. VINS: Visual Search for Mobile User Interface Design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1--14).Google ScholarDigital Library
- J. Chen, C. Chen, Z. Xing, X. Xia, L. Zhu, J. Grundy, J. Wang, 2020. Wireframe-based UI Design Search through Image Autoencoder. ACM Transactions on Software Engineering and Methodology, 29(3).Google Scholar
- X. Chen, Q. Zou, B. Fan, Z. Zheng, X. Luo, X, 2019. Recommending software features for mobile applications based on user interface comparison. Requirements Engineering, 24(4).Google Scholar
- J. Cohen, 1998. Statistical Power Analysis for the Behavioral Sciences. Routledge Academic.Google Scholar
- A. Cooper, R. Reimann e H. Dubberlyl, 2014. About face: the essentials of interaction design. Indianápolis: John Wiley and Sons.Google Scholar
- B. Deka, Z. Huang, C. Franzen, J. Hibschman, D. Afergan, Y. Li, J. Nichols, R. Kumar, 2017. Rico: A mobile app dataset for building data-driven design applications. Proc. of the 30st Annual ACM Symposium on User Interface Software and Technology, ACM.Google ScholarDigital Library
- X. Ge, 2019. Android GUI Search Using Hand-Drawn Sketches. Proc. of the 41st Int. Conference on Software Engineering, IEEE.Google ScholarDigital Library
- Y. Hu, G. Xu, B. Zhang, K. Lai, G. Xu, M. Zhang, 2020. Robust App Clone Detection Based on Similarity of UI Structure. IEEE Access, 8.Google Scholar
- F. Huang, J. F. Canny e J. Nichols, 2019. Swire: Sketch-based User Interface Retrieval. In Proc. of the CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, 1--10.Google ScholarDigital Library
- IBGE 2021. Pesquisa Nacional por Amostra de Domicílios (PNAD) Contínua 2018/2019. Instituto Brasileiro de Geografia e Estatística.Google Scholar
- Y. Kwon, 2022. YoloLabel. Disponível em: https://github.com/developer0hye/Yolo_Label. Acesso em: 10, mai, 2022.Google Scholar
- D. Manandhar, D. Ruta e J. Collomosse, 2020. Learning structural similarity of user interface layouts using graph networks. In European Conference on Computer Vision (pp. 730--746). Springer, Cham.Google ScholarDigital Library
- J. Mao, J. Bian, H. Ma, Y. Jia, Z. Liang e X. Jiang, 2018. Robust detection of android UI similarity. In 2018 IEEE International Conference on Communications (ICC) (pp. 1--6). IEEE.Google ScholarCross Ref
- MIT App Inventor. MIT App Inventor, 2022. Disponível em: <https://appinventor.mit.edu/>. Acesso em: 10, mai, 2022.Google Scholar
- J. Munkres, 1957. Algorithms for the Assignment and Transportation Problems. Journal of the Society for Industrial and Applied Mathematics, v. 5, n. 1, p. 32--38.Google Scholar
- E. Mustafaraj, F. Turbak, M. Svanberg, 2017. Identifying Original Projects in App Inventor. Proc. of the Florida Artificial Intelligence Research Society Conference, Marco Island, FL, EUA.Google Scholar
- T. T. Nguyen, P. M. Vu, H. V. Pham, T. T. Nguyen, 2018. Deep learning UI design patterns of mobile apps. Proc. of the 40th Int. Conference on Software Engineering, ACM.Google ScholarDigital Library
- V. P. S. Pandian e S. Suleri, 2020. BlackBox Toolkit: Intelligent Assistance to UI Design. Proc. of the Workshop on Artificial Intelligence for HCI: A Modern Approach, Honolulu, USA.Google Scholar
- B. D. Ripley, 1996. Pattern Recognition and Neural Networks. Cambridge University Press, Cambridge.Google ScholarDigital Library
- J. Royo, 2006. Design Digital. Coleção Fundamentos do Design. Tradução de Osvaldo Antônio Rosiano. 1.ed. São Paulo: Edições Rosari.Google Scholar
- A. S. de Souza, N. da C. Alves, C. Gresse von Wangenheim, L. Kreuch, 2021. Análise Automatizada da Originalidade de Design de Interfaces de Usuário no Contexto Educacional: Um Mapeamento da Literatura. In: Simpósio Brasileiro de Informática na Educação, 32, online. Porto Alegre: Sociedade Brasileira de Computação, p. 1140--1151. Google ScholarCross Ref
- C. Spearman, 1904. The Proof and Measurement of Association between Two Things. The American Journal of Psychology, v. 15, n. 1, p. 72--101.Google Scholar
- Statia, 2022. Number of apps available in leading app stores as of 1st quarter 2021. https://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/Google Scholar
- M. Svanberg, 2017. Using feature vector representations to identify similar projects in App Inventor. Proc. of the IEEE Blocks and Beyond Workshop, IEEE.Google ScholarCross Ref
- F. Turbak, E. Mustafaraj, M. Svanberg, M. Dawson, 2017. Work in Progress: identifying and analyzing original projects in an open-ended blocks programming environment. Proc. of the 23rd Int. Conference on Distributed Multimedia Systems, Pittsburgh/EUA.Google ScholarCross Ref
- Ultralytics, 2021. YOLOv5 Documentation, [s.d.]. Disponível em: https://docs.ultralytics.com/. Acesso em: 10, mai, 2021Google Scholar
- C. Wohlin et al, 2012. Experimentation in Software Engineering. Springer.Google Scholar
- Y. Xie, T. Lin, H. Xu, 2019. User Interface Code Retrieval: a novel visual-representation-aware approach. IEEE Access, 7.Google Scholar
- R. K. Yin, 2013. Case Study Research: Design and Methods SAGE Publications (5. ed.).Google Scholar
- Z. Zou, Z. Shi, Y. Guo e J. Ye, 2019. Object Detection in 20 Years: A Survey, arXiv:1905.05055v2 [cs.CV].Google Scholar
Index Terms
- Analyzing structural similarity of user interface layouts of Android apps using deep learning
Recommendations
Using Deep Learning to Support the User Interface Design of Mobile Applications with App Inventor
IHC '21: Proceedings of the XX Brazilian Symposium on Human Factors in Computing SystemsTransforming a user interface sketch into a wireframe using App Inventor has been a common need both in computing education and the development of applications by end users. As this part of mobile application development is challenging and time-...
Automated assessment of visual aesthetics of Android user interfaces with deep learning
IHC '22: Proceedings of the 21st Brazilian Symposium on Human Factors in Computing SystemsVisual aesthetics are seen as an essential success factor for mobile apps, affecting user experience and perception, which makes their evaluation crucial as part of the interface designing process. Machine learning approaches have shown to be very ...
The University of Alberta user interface management system
In this paper the design and implementation of the University of Alberta user interface management system (UIMS) is discussed. This UIMS is based on the Seeheim model of user interfaces, which divides the user interface into three separate components. ...
Comments