Analyzing structural similarity of user interface layouts of Android apps using deep learning
Resumo
The layout of Graphical user interfaces (GUIs) takes into consideration the distribution of buttons, images, texts, and other components that are important factors for a positive experience of interactivity and usability. Therefore, it may be helpful to consult similar layouts during the design of GUIs for inspiration as well as to direct the development of original interfaces. This may be even more important in the context of the design of mobile applications in an educational context or when being developed by end-users with low-code environments, such as App Inventor. Therefore, this article presents an approach for analyzing the similarity of GUI layout using deep learning. First, GUI components and their positions are detected using the YOLO model. Then, the similarity of layouts is analyzed based on the GUI components detected. Results of the evaluation of the developed approach indicated a good accuracy of the detection of GUI components and the analysis of similarity close to human predictions. The results of this research can be used for the automated analysis of the similarity of GUI layouts of mobile applications, which can be used to analyze the design of mobile applications (consistency), search for similar designs, or assess the originality of a GUI design. It can also be applied in an educational context to assess the development of creativity as part of the development of mobile applications in order to teach computing.
Palavras-chave:
Mobile Application, User Interface Design, Similarity, App Inventor, Deep Learning
Referências
S. Almrayat, R. Yousef e A. Sharieh, 2019. Evaluating the Impact of GUI Similarity between Android Applications to Measure their Functional Similarity. International Journal of Computer Applications, 975, 8887.
G. Alves, 2020. Detecção de Objetos com YOLO - Uma abordagem moderna. In: Expert Academy. [S.I.], 13 out. 2020. Disponível em: [link]. Acesso em: 10 mai. 2022.
N. da C. Alves, C. Gresse von Wangenheim, J. C. R. Hauck, 2020. Teaching Programming to Novices: A Large-scale Analysis of App Inventor Projects. In: Proc. of the 15th Latin-American Conference on Learning Technologies, Loja/Ecuador.
S. Amershi, et al, 2019. Software engineering for machine learning: A case study. International Conference on Software Engineering: Software Engineering in Practice. Montreal, QC, Canada: IEEE. p. 291--300.
D. d. S. Baulé, C. Gresse von Wangenheim, A. von Wangenheim, J. C. R. Hauck, 2020. Recent Progress in Automated Code Generation from GUI Images using Machine Learning Techniques. Journal of Universal Computer Science, 26(9).
D. d. S. Baulé, C. Gresse von Wangenheim, J. C. R. Hauck, A. von Wangenheim, E. C. Vargas Júnior, 2021. Using Deep Learning to Support the User Interface Design of Mobile Applications with App Inventor. In Anais do XX Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais, online.
F. Behrang, S. P. Reiss, A. Orso, 2018. GUIfetch. Proc. of the 5th Int. Conference on Mobile Software Engineering and Systems, ACM, EUA.
S. Bunian, K. Li, C. Jemmali, C. Harteveld, Y. Fu, e M. S. Seif El-Nasr, 2021. VINS: Visual Search for Mobile User Interface Design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1--14).
J. Chen, C. Chen, Z. Xing, X. Xia, L. Zhu, J. Grundy, J. Wang, 2020. Wireframe-based UI Design Search through Image Autoencoder. ACM Transactions on Software Engineering and Methodology, 29(3).
X. Chen, Q. Zou, B. Fan, Z. Zheng, X. Luo, X, 2019. Recommending software features for mobile applications based on user interface comparison. Requirements Engineering, 24(4).
J. Cohen, 1998. Statistical Power Analysis for the Behavioral Sciences. Routledge Academic.
A. Cooper, R. Reimann e H. Dubberlyl, 2014. About face: the essentials of interaction design. Indianápolis: John Wiley and Sons.
B. Deka, Z. Huang, C. Franzen, J. Hibschman, D. Afergan, Y. Li, J. Nichols, R. Kumar, 2017. Rico: A mobile app dataset for building data-driven design applications. Proc. of the 30st Annual ACM Symposium on User Interface Software and Technology, ACM.
X. Ge, 2019. Android GUI Search Using Hand-Drawn Sketches. Proc. of the 41st Int. Conference on Software Engineering, IEEE.
Y. Hu, G. Xu, B. Zhang, K. Lai, G. Xu, M. Zhang, 2020. Robust App Clone Detection Based on Similarity of UI Structure. IEEE Access, 8.
F. Huang, J. F. Canny e J. Nichols, 2019. Swire: Sketch-based User Interface Retrieval. In Proc. of the CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, 1--10.
IBGE 2021. Pesquisa Nacional por Amostra de Domicílios (PNAD) Contínua 2018/2019. Instituto Brasileiro de Geografia e Estatística.
Y. Kwon, 2022. YoloLabel. Disponível em: https://github.com/developer0hye/Yolo_Label. Acesso em: 10, mai, 2022.
D. Manandhar, D. Ruta e J. Collomosse, 2020. Learning structural similarity of user interface layouts using graph networks. In European Conference on Computer Vision (pp. 730--746). Springer, Cham.
J. Mao, J. Bian, H. Ma, Y. Jia, Z. Liang e X. Jiang, 2018. Robust detection of android UI similarity. In 2018 IEEE International Conference on Communications (ICC) (pp. 1--6). IEEE.
MIT App Inventor. MIT App Inventor, 2022. Disponível em: <https://appinventor.mit.edu/>. Acesso em: 10, mai, 2022.
J. Munkres, 1957. Algorithms for the Assignment and Transportation Problems. Journal of the Society for Industrial and Applied Mathematics, v. 5, n. 1, p. 32--38.
E. Mustafaraj, F. Turbak, M. Svanberg, 2017. Identifying Original Projects in App Inventor. Proc. of the Florida Artificial Intelligence Research Society Conference, Marco Island, FL, EUA.
T. T. Nguyen, P. M. Vu, H. V. Pham, T. T. Nguyen, 2018. Deep learning UI design patterns of mobile apps. Proc. of the 40th Int. Conference on Software Engineering, ACM.
V. P. S. Pandian e S. Suleri, 2020. BlackBox Toolkit: Intelligent Assistance to UI Design. Proc. of the Workshop on Artificial Intelligence for HCI: A Modern Approach, Honolulu, USA.
B. D. Ripley, 1996. Pattern Recognition and Neural Networks. Cambridge University Press, Cambridge.
J. Royo, 2006. Design Digital. Coleção Fundamentos do Design. Tradução de Osvaldo Antônio Rosiano. 1.ed. São Paulo: Edições Rosari.
A. S. de Souza, N. da C. Alves, C. Gresse von Wangenheim, L. Kreuch, 2021. Análise Automatizada da Originalidade de Design de Interfaces de Usuário no Contexto Educacional: Um Mapeamento da Literatura. In: Simpósio Brasileiro de Informática na Educação, 32, online. Porto Alegre: Sociedade Brasileira de Computação, p. 1140--1151.
C. Spearman, 1904. The Proof and Measurement of Association between Two Things. The American Journal of Psychology, v. 15, n. 1, p. 72--101.
Statia, 2022. Number of apps available in leading app stores as of 1st quarter 2021. [link].
M. Svanberg, 2017. Using feature vector representations to identify similar projects in App Inventor. Proc. of the IEEE Blocks and Beyond Workshop, IEEE.
F. Turbak, E. Mustafaraj, M. Svanberg, M. Dawson, 2017. Work in Progress: identifying and analyzing original projects in an open-ended blocks programming environment. Proc. of the 23rd Int. Conference on Distributed Multimedia Systems, Pittsburgh/EUA.
Ultralytics, 2021. YOLOv5 Documentation, [s.d.]. Disponível em: https://docs.ultralytics.com/. Acesso em: 10, mai, 2021
C. Wohlin et al, 2012. Experimentation in Software Engineering. Springer.
Y. Xie, T. Lin, H. Xu, 2019. User Interface Code Retrieval: a novel visual-representation-aware approach. IEEE Access, 7.
R. K. Yin, 2013. Case Study Research: Design and Methods SAGE Publications (5. ed.).
Z. Zou, Z. Shi, Y. Guo e J. Ye, 2019. Object Detection in 20 Years: A Survey, arXiv:1905.05055v2 [cs.CV].
G. Alves, 2020. Detecção de Objetos com YOLO - Uma abordagem moderna. In: Expert Academy. [S.I.], 13 out. 2020. Disponível em: [link]. Acesso em: 10 mai. 2022.
N. da C. Alves, C. Gresse von Wangenheim, J. C. R. Hauck, 2020. Teaching Programming to Novices: A Large-scale Analysis of App Inventor Projects. In: Proc. of the 15th Latin-American Conference on Learning Technologies, Loja/Ecuador.
S. Amershi, et al, 2019. Software engineering for machine learning: A case study. International Conference on Software Engineering: Software Engineering in Practice. Montreal, QC, Canada: IEEE. p. 291--300.
D. d. S. Baulé, C. Gresse von Wangenheim, A. von Wangenheim, J. C. R. Hauck, 2020. Recent Progress in Automated Code Generation from GUI Images using Machine Learning Techniques. Journal of Universal Computer Science, 26(9).
D. d. S. Baulé, C. Gresse von Wangenheim, J. C. R. Hauck, A. von Wangenheim, E. C. Vargas Júnior, 2021. Using Deep Learning to Support the User Interface Design of Mobile Applications with App Inventor. In Anais do XX Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais, online.
F. Behrang, S. P. Reiss, A. Orso, 2018. GUIfetch. Proc. of the 5th Int. Conference on Mobile Software Engineering and Systems, ACM, EUA.
S. Bunian, K. Li, C. Jemmali, C. Harteveld, Y. Fu, e M. S. Seif El-Nasr, 2021. VINS: Visual Search for Mobile User Interface Design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1--14).
J. Chen, C. Chen, Z. Xing, X. Xia, L. Zhu, J. Grundy, J. Wang, 2020. Wireframe-based UI Design Search through Image Autoencoder. ACM Transactions on Software Engineering and Methodology, 29(3).
X. Chen, Q. Zou, B. Fan, Z. Zheng, X. Luo, X, 2019. Recommending software features for mobile applications based on user interface comparison. Requirements Engineering, 24(4).
J. Cohen, 1998. Statistical Power Analysis for the Behavioral Sciences. Routledge Academic.
A. Cooper, R. Reimann e H. Dubberlyl, 2014. About face: the essentials of interaction design. Indianápolis: John Wiley and Sons.
B. Deka, Z. Huang, C. Franzen, J. Hibschman, D. Afergan, Y. Li, J. Nichols, R. Kumar, 2017. Rico: A mobile app dataset for building data-driven design applications. Proc. of the 30st Annual ACM Symposium on User Interface Software and Technology, ACM.
X. Ge, 2019. Android GUI Search Using Hand-Drawn Sketches. Proc. of the 41st Int. Conference on Software Engineering, IEEE.
Y. Hu, G. Xu, B. Zhang, K. Lai, G. Xu, M. Zhang, 2020. Robust App Clone Detection Based on Similarity of UI Structure. IEEE Access, 8.
F. Huang, J. F. Canny e J. Nichols, 2019. Swire: Sketch-based User Interface Retrieval. In Proc. of the CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, 1--10.
IBGE 2021. Pesquisa Nacional por Amostra de Domicílios (PNAD) Contínua 2018/2019. Instituto Brasileiro de Geografia e Estatística.
Y. Kwon, 2022. YoloLabel. Disponível em: https://github.com/developer0hye/Yolo_Label. Acesso em: 10, mai, 2022.
D. Manandhar, D. Ruta e J. Collomosse, 2020. Learning structural similarity of user interface layouts using graph networks. In European Conference on Computer Vision (pp. 730--746). Springer, Cham.
J. Mao, J. Bian, H. Ma, Y. Jia, Z. Liang e X. Jiang, 2018. Robust detection of android UI similarity. In 2018 IEEE International Conference on Communications (ICC) (pp. 1--6). IEEE.
MIT App Inventor. MIT App Inventor, 2022. Disponível em: <https://appinventor.mit.edu/>. Acesso em: 10, mai, 2022.
J. Munkres, 1957. Algorithms for the Assignment and Transportation Problems. Journal of the Society for Industrial and Applied Mathematics, v. 5, n. 1, p. 32--38.
E. Mustafaraj, F. Turbak, M. Svanberg, 2017. Identifying Original Projects in App Inventor. Proc. of the Florida Artificial Intelligence Research Society Conference, Marco Island, FL, EUA.
T. T. Nguyen, P. M. Vu, H. V. Pham, T. T. Nguyen, 2018. Deep learning UI design patterns of mobile apps. Proc. of the 40th Int. Conference on Software Engineering, ACM.
V. P. S. Pandian e S. Suleri, 2020. BlackBox Toolkit: Intelligent Assistance to UI Design. Proc. of the Workshop on Artificial Intelligence for HCI: A Modern Approach, Honolulu, USA.
B. D. Ripley, 1996. Pattern Recognition and Neural Networks. Cambridge University Press, Cambridge.
J. Royo, 2006. Design Digital. Coleção Fundamentos do Design. Tradução de Osvaldo Antônio Rosiano. 1.ed. São Paulo: Edições Rosari.
A. S. de Souza, N. da C. Alves, C. Gresse von Wangenheim, L. Kreuch, 2021. Análise Automatizada da Originalidade de Design de Interfaces de Usuário no Contexto Educacional: Um Mapeamento da Literatura. In: Simpósio Brasileiro de Informática na Educação, 32, online. Porto Alegre: Sociedade Brasileira de Computação, p. 1140--1151.
C. Spearman, 1904. The Proof and Measurement of Association between Two Things. The American Journal of Psychology, v. 15, n. 1, p. 72--101.
Statia, 2022. Number of apps available in leading app stores as of 1st quarter 2021. [link].
M. Svanberg, 2017. Using feature vector representations to identify similar projects in App Inventor. Proc. of the IEEE Blocks and Beyond Workshop, IEEE.
F. Turbak, E. Mustafaraj, M. Svanberg, M. Dawson, 2017. Work in Progress: identifying and analyzing original projects in an open-ended blocks programming environment. Proc. of the 23rd Int. Conference on Distributed Multimedia Systems, Pittsburgh/EUA.
Ultralytics, 2021. YOLOv5 Documentation, [s.d.]. Disponível em: https://docs.ultralytics.com/. Acesso em: 10, mai, 2021
C. Wohlin et al, 2012. Experimentation in Software Engineering. Springer.
Y. Xie, T. Lin, H. Xu, 2019. User Interface Code Retrieval: a novel visual-representation-aware approach. IEEE Access, 7.
R. K. Yin, 2013. Case Study Research: Design and Methods SAGE Publications (5. ed.).
Z. Zou, Z. Shi, Y. Guo e J. Ye, 2019. Object Detection in 20 Years: A Survey, arXiv:1905.05055v2 [cs.CV].
Publicado
17/10/2022
Como Citar
DA CRUZ ALVES, Nathalia; KREUCH, Leonardo; GRESSE VON WANGENHEIM, Christiane.
Analyzing structural similarity of user interface layouts of Android apps using deep learning. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 21. , 2022, Diamantina.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
