José Aldo Silva Da Costa
ABSTRACT
Code comprehension is essential for software evolution, but it can be hindered by structural problems in the code. Despite its importance, researchers often rely on static code metrics to investigate code comprehension, without considering dynamic aspects such as the visual effort involved. Therefore, we aim to fill this gap by examining code comprehension using eye-tracking metrics. We conducted three controlled eye-tracking studies with a total of 128 novices in three distinct programming languages: C, Python, and Java. We used constructions with different granularity levels, namely atoms of confusion, refactorings, and #ifdef annotations. With respect to atoms of confusion, we found that the clarified version of the code reduced the time and number of attempts by 38.6% and 28%, respectively. The use of the Extract Method refactoring significantly reduced the time of two tasks by 70% to 78.8%, while increasing the accuracy of three tasks by 20% to 34.4%. The use of #ifdef annotations presented reductions in the modified region by 46.9% in time, 44.7% in the fixation duration, 48.4% in the fixation count, and 60.5% in the regressions count. In addition, we also identified patterns in our data by observing several visual transitions going forward and backward between specific code elements which allowed us to infer confusion in the code, such as when adding the particle not before the variable. These results provide educators, practitioners, and researchers with valuable information on certain transformations and their potential to ease or hinder code comprehension. The study also highlights the potential of visual metrics to reveal the impact of transformations that cannot be captured by static code metrics. Our results also pave the way for the development of more advanced tools that track the eye movements of developers and assist them with helpful suggestions.
- Dave Binkley, Marcia Davis, Dawn Lawrie, Jonathan Maletic, Christopher Morrell, and Bonita Sharif. 2013. The Impact of Identifier Style on Effort and Comprehension. Empirical Software Engineering 18, 2 (2013), 219–276.Google Scholar
Digital Library
- Teresa Busjahn, Carsten Schulte, and Andreas Busjahn. 2011. Analysis of Code Reading to Gain More Insight in Program Comprehension(Koli Calling’11). 1–9.Google Scholar
- Fernando Castor. 2018. Identifying Confusing Code in Swift Programs(VEM’18).Google Scholar
- Diego Cedrim, Alessandro Garcia, Melina Mongiovi, Rohit Gheyi, Leonardo Sousa, Rafael de Mello, Baldoino Fonseca, Márcio Ribeiro, and Alexander Chávez. 2017. Understanding the Impact of Refactoring on Smells: A Longitudinal Study of 23 Software Projects(ESEC/FSE’17). 465–475.Google Scholar
- José Aldo Silva da Costa, Rohit Gheyi, Fernando Castor, Pablo Roberto Fernandes de Oliveira, Márcio Ribeiro, and Baldoino Fonseca. 2023. Seeing Confusion through a New Lens: on the Impact of Atoms of Confusion on Novices’ Code Comprehension. Empirical Software Engineering 28, 4 (2023), 81.Google ScholarDigital Library
- José Aldo Silva da Costa, Rohit Gheyi, Márcio Ribeiro, Sven Apel, Vander Alves, Baldoino Fonseca, Flávio Medeiros, and Alessandro Garcia. 2021. Evaluating Refactorings for Disciplining #ifdef Annotations: An Eye Tracking Study with Novices. Empirical Software Engineering 26, 5 (2021), 1–35.Google Scholar
- da Costa et al.2023. “Supplementary Material for the Thesis (Artifacts)”. (2023). At https://github.com/josealdo/material-complementar-tese.Google Scholar
- Benedito de Oliveira, Márcio Ribeiro, José Aldo Silva da Costa, Rohit Gheyi, Guilherme Amaral, Rafael de Mello, Anderson Oliveira, Alessandro Garcia, Rodrigo Bonifácio, and Baldoino Fonseca. 2020. Atoms of Confusion: The Eyes Do Not Lie(SBES’20). 243–252.Google Scholar
- Wolfram Fenske, Jacob Krüger, Maria Kanyshkova, and Sandro Schulze. 2020. #ifdef Directives and Program Comprehension: The Dilemma between Correctness and Preference(ICSME’20).Google Scholar
- Martin Fowler. 1999. Refactoring: Improving the Design of Existing Code. Boston, MA, USA: Addison-Wesley/Longman.Google ScholarDigital Library
- Dan Gopstein, Anne-Laure Fayard, Sven Apel, and Justin Cappos. 2020. Thinking Aloud about Confusing Code: A Qualitative Investigation of Program Comprehension and Atoms of Confusion(ESEC/FSE’20). 605–616.Google Scholar
- Dan Gopstein, Jake Iannacone, Yu Yan, Lois DeLong, Yanyan Zhuang, Martin K.-C. Yeh, and Justin Cappos. 2017. Understanding Misunderstandings in Source Code(ESEC/FSE’17). 129–139.Google Scholar
- Dan Gopstein, Hongwei Henry Zhou, Phyllis Frankl, and Justin Cappos. 2018. Prevalence of Confusing Code in Software Projects: Atoms of Confusion in the Wild(ICSMR’18). 281–291.Google Scholar
- Chris Langhout and Maurício Aniche. 2021. Atoms of Confusion in Java((ICPC’21)). IEEE, 25–35.Google Scholar
- Jörg Liebig, Christian Kästner, and Sven Apel. 2011. Analyzing the discipline of preprocessor annotations in 30 million lines of C code(AOSD’11). 191–202.Google Scholar
- Romero Malaquias, Márcio Ribeiro, Rodrigo Bonifácio, Eduardo Monteiro, Flávio Medeiros, Alessandro Garcia, and Rohit Gheyi. 2017. The Discipline of Preprocessor-Based Annotations – Does #ifdef TAG n’t #endif Matter(ICPC’17). IEEE, 297–307.Google Scholar
- Flávio Medeiros, Gabriel Lima, Guilherme Amaral, Sven Apel, Christian Kästner, Márcio Ribeiro, and Rohit Gheyi. 2019. An Investigation of Misunderstanding Code Patterns in C Open-source Software Projects. Empirical Software Engineering 24, 4 (2019), 1693–1726.Google ScholarDigital Library
- Flávio Medeiros, Márcio Ribeiro, Rohit Gheyi, Sven Apel, Christian Kästner, Bruno Ferreira, Luiz Carvalho, and Baldoino Fonseca. 2018. Discipline matters: Refactoring of preprocessor directives in the #ifdef hell. IEEE Transactions on Software Engineering 44, 5 (2018), 453–469.Google ScholarCross Ref
- Flávio Medeiros, Iran Rodrigues, Márcio Ribeiro, Leopoldo Teixeira, and Rohit Gheyi. 2015. An empirical study on configuration-related issues: investigating undeclared and unused identifiers. In (GPCE’15). 35–44.Google Scholar
- Wendell Mendes, Windson Viana, and Lincoln Rocha. 2021. BOHR - Uma Ferramenta para a Identificação de Átomos de Confusão em Códigos Java(VEM’21). SBC, 41–45.Google Scholar
- Melina Mongiovi, Rohit Gheyi, Gustavo Soares, Márcio Ribeiro, Paulo Borba, and Leopoldo Teixeira. 2018. Detecting Overly Strong Preconditions in Refactoring Engines. IEEE Transactions on Software Engineering 44, 5 (2018), 429–452.Google ScholarCross Ref
- Sandro Schulze, Jörg Liebig, Janet Siegmund, and Sven Apel. 2013. Does the Discipline of Preprocessor Annotations Matter?: A Controlled Experiment(GPCE ’13). 65–74.Google Scholar
- Zohreh Sharafi, Timothy Shaffer, Bonita Sharif, and Yann-Gaël Guéhéneuc. 2015. Eye-tracking Metrics in Software Engineering(APSEC’15). IEEE, 96–103.Google Scholar
- Bonita Sharif, Michael Falcone, and Jonathan Maletic. 2012. An Eye-tracking Study on the Role of Scan Time in Finding Source Code Defects(ETRA’12). ACM, 381–384.Google Scholar
- Bonita Sharif and Jonathan Maletic. 2010. An Eye Tracking Study on Camelcase and Under_score Identifier Styles(ICPC’10). IEEE, 196–205.Google Scholar
- Gustavo Soares, Rohit Gheyi, and Tiago Massoni. 2013. Automated Behavioral Testing of Refactoring Engines. IEEE Transactions on Software Engineering 39, 2 (2013), 147–162.Google ScholarDigital Library
- Anselm Strauss and Juliet Corbin. 1998. Basics of Qualitative Research Techniques. Thousand Oaks, CA: Sage publications.Google Scholar
- Martin K-C Yeh, Yu Yan, Yanyan Zhuang, and Lois Anne DeLong. 2021. Identifying Program Confusion Using Electroencephalogram Measurements. Behaviour & Information Technology (2021), 1–18.Google Scholar
Index Terms
- Evaluating the Code Comprehension of Novices with Eye Tracking
Recommendations
Eye tracking and program comprehension
EMIP '19: Proceedings of the 6th International Workshop on Eye Movements in ProgrammingReading and comprehending code is different from reading regular text. And eye tracking, which measures where we look, for how long, and how much mental effort we exert, can provide crucial data. The talk will survey some achievements and suggest future ...
Analysis of code reading to gain more insight in program comprehension
Koli Calling '11: Proceedings of the 11th Koli Calling International Conference on Computing Education ResearchCode reading, although an integral part of program comprehension, is rarely reflected. In this paper, we want to argue for a research approach and direction exploiting the potential that lies in the analysis of reading processes. Based on the vast ...
Analyzing the influence of block highlighting on beginning programmers' reading behavior using eye tracking
CSERC '20: Proceedings of the 9th Computer Science Education Research ConferenceTo improve programming education, it is worthwhile to investigate how understanding of source code takes place. Reading is a foundational skill essential to tracing, debugging and understanding code. As readers' working memory is limited, their ...
Comments