Investigating Cognitive Demands in Software Testing Activities: An EEG-Based Experimental Study
Resumo
Software testing encompasses a variety of cognitively demanding activities, including code comprehension, fault identification, and test case design. Understanding the cognitive demands associated with these activities is important for improving testing processes, training programs, and tool support. However, despite the central role of software testers in software quality assurance, the cognitive processes underlying testing activities remain relatively underexplored. In this paper,we investigate neural activity and performance during software testing using electroencephalography (EEG). We conducted a controlled experiment with eight professional software testers performing four common testing activities: test code comprehension, syntax error identification, logic error identification, and test case design. Our results provide descriptive evidence of variation in neural activity across software testing activities, although the Friedman tests did not reveal statistically significant differences among activities. Logic Error Identification produced the largest reduction in Alpha activity and the lowest participant accuracy, suggesting greater attentional and analytical demands. Test Case Design, meanwhile, exhibited the highest Theta and Delta activity levels while maintaining relatively high performance, suggesting greater engagement of cognitive resources, including working memory. Furthermore, a strong positive association was observed between Theta activity and performance during Logic Error Identification (𝜌 = 0.77, 𝑝 = 0.025). Overall, these findings provide preliminary evidence that software testing activities may involve different cognitive demands, motivating further investigation into how cognitive insights can inform testing practices, training strategies, automation techniques, and AI-assisted testing tools.
Palavras-chave:
Software Testing, Cognitive Load, Electroencephalography (EEG), Human Factors, Software Engineering
Referências
Pavlo Antonenko, Fred Paas, Roland Grabner, and Tamara Van Gog. 2010. Using electroencephalography to measure cognitive load. Educational psychology review 22 (2010), 425–438.
Neil C. C. Brown, Pierre Weill-Tessier, Maksymilian Sekula, Alexandra-Lucia Costache, and Michael Kölling. 2022. Novice Use of the Java Programming Language. ACM Trans. Comput. Educ. 23, 1, Article 10 (dec 2022), 24 pages.
Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye movements in code reading: Relaxing the linear order. In 2015 ieee 23rd international conference on program comprehension. IEEE, 255–265.
Alessandra Calcagno, Stefania Coelli, Ricardo Couceiro, João Durães, Caterina Amendola, Ileana Pirovano, Rebecca Re, and Anna Maria Bianchi. 2020. EEG monitoring during software development. In 2020 IEEE 20th Mediterranean Electrotechnical Conference (MELECON). IEEE, Palermo, Italy, 325–329.
Camila Cosmo, Amin Zandvakili, Nicholas J. Petrosino, Thaise Graziele L. de O. Toutain, José Garcia Vivas Miranda, and Noah S. Philip. 2023. Examining the neural mechanisms of rTMS: a naturalistic pilot study of acute and serial effects in pharmacoresistant depression.
Igor Crk, Timothy Kluthe, and Andreas Stefik. 2015. Understanding programming expertise: an empirical study of phasic brain wave changes. ACM Transactions on Computer-Human Interaction (TOCHI) 23, 1 (2015), 1–29.
Madeline Endres, Zachary Karas, Xiaosu Hu, Ioulia Kovelman, and Westley Weimer. 2021. Relating Reading, Visualization, and Coding for New Programmers: A Neuroimaging Study. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, Madrid, ES, 600–612.
Davide Fucci, Daniela Girardi, Nicole Novielli, Luigi Quaranta, and Filippo Lanubile. 2019. A replication study on code comprehension and expertise using lightweight biometric sensors. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, Montreal, QC, Canada, 311–322.
William S. Greenwell, Elisabeth A. Strunk, and John C. Knight. 2004. Failure Analysis and the Safety-Case Lifecycle. In Human Error, Safety and Systems Development, ChrisW. Johnson and Philippe Palanque (Eds.). Springer US, Boston, MA, 163–176.
Fatih Gurcan, Gonca Gokce Menekse Dalveren, Nergiz Ercil Cagiltay, Dumitru Roman, and Ahmet Soylu. 2022. Evolution of Software Testing Strategies and Trends: Semantic Content Analysis of Software Research Corpus of the Last 40 Years. IEEE Access 10 (2022), 106093–106109.
Thalía Harmony. 2013. The functional significance of theta oscillations in cognitive processing. Frontiers in Integrative Neuroscience 7 (2013), 58.
Yoshiharu Ikutani and Hidetake Uwano. 2014. Brain activity measurement during program comprehension with NIRS. In 15th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD). IEEE, Las Vegas, NV, USA, 1–6.
Wolfgang Klimesch. 1999. EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain research reviews 29, 2-3 (1999), 169–195.
Ryan Krueger, Yu Huang, Xinyu Liu, Tyler Santander,WestleyWeimer, and Kevin Leach. 2020. Neurological divide: an fMRI study of prose and code writing. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE ’20). Association for Computing Machinery, New York, NY, USA, 678–690.
SeolHwa Lee, Andrew Matteson, Danial Hooshyar, SongHyun Kim, JaeBum Jung, GiChun Nam, and HeuiSeok Lim. 2016. Comparing programming language comprehension between novice and expert programmers using eeg analysis. In 2016 IEEE 16th international conference on bioinformatics and bioengineering (BIBE). IEEE, Taichung, Taiwan, 350–355.
Yu-Tzu Lin, Yi-Zhi Liao, Xiao Hu, and Cheng-Chih Wu. 2021. Eeg activities during program comprehension: An exploration of cognition. IEEE Access 9 (2021), 120407–120421.
Sergio Machado, Cláudio Portella, Julio Silva, Bruna Velasques, Patrícia Terra, Camila Ferreira-Vorkapic, Vernon Silva, Luis Miana, Luis Basile, Mauricio Cagy, Roberto Piedade, and Pedro Ribeiro. 2007. Changes in quantitative EEG absolute power during the task of catching an object in free fall. Arquivos de neuropsiquiatria 65 (10 2007), 633–6.
Christoph Mayr-Dorn, Michael Vierhauser, Stefan Bichler, Felix Keplinger, Jane Cleland-Huang, Alexander Egyed, and Thomas Mehofer. 2021. Supporting quality assurance with automated process-centric quality constraints checking. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, Madrid, ES, 1298–1310.
Júlio Medeiros, Ricardo Couceiro, Gonçalo Duarte, João Durães, João Castelhano, Catarina Duarte, Miguel Castelo-Branco, Henrique Madeira, Paulo de Carvalho, and César Teixeira. 2021. Can EEG be adopted as a neuroscience reference for assessing software programmers’ cognitive load? Sensors 21, 7 (2021), 2338.
J. Medeiros, M. Simões, J. Castelhano, R. Abreu, R. Couceiro, J. Henriques, et al. 2024. EEG as a potential ground truth for the assessment of cognitive state in software development activities: A multimodal imaging study. PLoS ONE 19, 3 (2024), e0299108.
Norman Peitek, Sven Apel, Chris Parnin, Andre Brechmann, and Janet Siegmund. 2021. Program comprehension and code complexity metrics: An fMRI study. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) (Madrid, Spain). IEEE, Madrid, Spain, 524–536.
Norman Peitek, Annabelle Bergum, Maurice Rekrut, Jonas Mucke, Matthias Nadig, Chris Parnin, Janet Siegmund, and Sven Apel. 2022. Correlates of programmer efficacy and their link to experience: a combined EEG and eye-tracking study. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore, Singapore) (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 120–131.
Norman Peitek, Janet Siegmund, Sven Apel, Christian Kästner, Chris Parnin, Anja Bethmann, Thomas Leich, Gunter Saake, and André Brechmann. 2018. A look into programmers’ heads. IEEE Transactions on Software Engineering 46, 4 (2018), 442–462.
Norman Peitek, Janet Siegmund, Chris Parnin, Sven Apel, Johannes C. Hofmeister, and André Brechmann. 2018. Simultaneous measurement of program comprehension with fMRI and eye tracking: a case study. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (Oulu, Finland) (ESEM ’18). Association for Computing Machinery, New York, NY, USA, Article 24, 10 pages.
R. Pressman. 2014. Software Engineering: A Practitioner’s Approach. McGraw-Hill Education. [link]
Patrick Rempel, Patrick Mäder, Tobias Kuschke, and Jane Cleland-Huang. 2014. Mind the gap: assessing the conformance of software traceability to relevant guidelines. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 943–954.
Saeid Sanei and J.A. Chambers. 2007. Introduction to EEG. John Wiley & Sons, Ltd, Chapter 1, 1–34.
Samuel Sanford Shapiro and Martin B Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika 52, 3-4 (1965), 591–611.
Siegel Sidney. 1957. Nonparametric statistics for the behavioral sciences. The Journal of Nervous and Mental Disease 125, 3 (1957), 497.
Janet Siegmund, André Brechmann, Sven Apel, Christian Kästner, Jörg Liebig, Thomas Leich, and Gunter Saake. 2012. Toward measuring program comprehension with functional magnetic resonance imaging. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (Cary, North Carolina) (FSE ’12). Association for Computing Machinery, New York, NY, USA, Article 24, 4 pages.
Janet Siegmund, Christian Kästner, Sven Apel, Chris Parnin, Anja Bethmann, Thomas Leich, Gunter Saake, and André Brechmann. 2014. Understanding understanding source code with functional magnetic resonance imaging. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 378–389.
Janet Siegmund, Christian Kästner, Sven Apel, André Brechmann, and Gunter Saake. 2013. Experience from measuring program comprehension - toward a general framework.
Janet Siegmund, Norman Peitek, Chris Parnin, Sven Apel, Johannes Hofmeister, Christian Kästner, Andrew Begel, Anja Bethmann, and André Brechmann. 2017. Measuring neural efficiency of program comprehension. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 140–150.
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Transactions on Software Engineering 50, 4 (2024), 911–936.
Barbara Weber, Thomas Fischer, and René Riedl. 2021. Brain and autonomic nervous system activity measurement in software engineering: A systematic literature review. Journal of Systems and Software 178 (2021), 110946.
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
Marvin Wyrich, Justus Bogner, and Stefan Wagner. 2023. 40 years of designing code comprehension experiments: A systematic mapping study. Comput. Surveys 56, 4 (2023), 1–42.
Martin K-C Yeh, Dan Gopstein, Yu Yan, and Yanyan Zhuang. 2017. Detecting and comparing brain activity in short program comprehension using EEG. In 2017 IEEE Frontiers in Education Conference (FIE). IEEE, Indianapolis, USA, 1–5.
Neil C. C. Brown, Pierre Weill-Tessier, Maksymilian Sekula, Alexandra-Lucia Costache, and Michael Kölling. 2022. Novice Use of the Java Programming Language. ACM Trans. Comput. Educ. 23, 1, Article 10 (dec 2022), 24 pages.
Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye movements in code reading: Relaxing the linear order. In 2015 ieee 23rd international conference on program comprehension. IEEE, 255–265.
Alessandra Calcagno, Stefania Coelli, Ricardo Couceiro, João Durães, Caterina Amendola, Ileana Pirovano, Rebecca Re, and Anna Maria Bianchi. 2020. EEG monitoring during software development. In 2020 IEEE 20th Mediterranean Electrotechnical Conference (MELECON). IEEE, Palermo, Italy, 325–329.
Camila Cosmo, Amin Zandvakili, Nicholas J. Petrosino, Thaise Graziele L. de O. Toutain, José Garcia Vivas Miranda, and Noah S. Philip. 2023. Examining the neural mechanisms of rTMS: a naturalistic pilot study of acute and serial effects in pharmacoresistant depression.
Igor Crk, Timothy Kluthe, and Andreas Stefik. 2015. Understanding programming expertise: an empirical study of phasic brain wave changes. ACM Transactions on Computer-Human Interaction (TOCHI) 23, 1 (2015), 1–29.
Madeline Endres, Zachary Karas, Xiaosu Hu, Ioulia Kovelman, and Westley Weimer. 2021. Relating Reading, Visualization, and Coding for New Programmers: A Neuroimaging Study. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, Madrid, ES, 600–612.
Davide Fucci, Daniela Girardi, Nicole Novielli, Luigi Quaranta, and Filippo Lanubile. 2019. A replication study on code comprehension and expertise using lightweight biometric sensors. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, Montreal, QC, Canada, 311–322.
William S. Greenwell, Elisabeth A. Strunk, and John C. Knight. 2004. Failure Analysis and the Safety-Case Lifecycle. In Human Error, Safety and Systems Development, ChrisW. Johnson and Philippe Palanque (Eds.). Springer US, Boston, MA, 163–176.
Fatih Gurcan, Gonca Gokce Menekse Dalveren, Nergiz Ercil Cagiltay, Dumitru Roman, and Ahmet Soylu. 2022. Evolution of Software Testing Strategies and Trends: Semantic Content Analysis of Software Research Corpus of the Last 40 Years. IEEE Access 10 (2022), 106093–106109.
Thalía Harmony. 2013. The functional significance of theta oscillations in cognitive processing. Frontiers in Integrative Neuroscience 7 (2013), 58.
Yoshiharu Ikutani and Hidetake Uwano. 2014. Brain activity measurement during program comprehension with NIRS. In 15th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD). IEEE, Las Vegas, NV, USA, 1–6.
Wolfgang Klimesch. 1999. EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain research reviews 29, 2-3 (1999), 169–195.
Ryan Krueger, Yu Huang, Xinyu Liu, Tyler Santander,WestleyWeimer, and Kevin Leach. 2020. Neurological divide: an fMRI study of prose and code writing. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE ’20). Association for Computing Machinery, New York, NY, USA, 678–690.
SeolHwa Lee, Andrew Matteson, Danial Hooshyar, SongHyun Kim, JaeBum Jung, GiChun Nam, and HeuiSeok Lim. 2016. Comparing programming language comprehension between novice and expert programmers using eeg analysis. In 2016 IEEE 16th international conference on bioinformatics and bioengineering (BIBE). IEEE, Taichung, Taiwan, 350–355.
Yu-Tzu Lin, Yi-Zhi Liao, Xiao Hu, and Cheng-Chih Wu. 2021. Eeg activities during program comprehension: An exploration of cognition. IEEE Access 9 (2021), 120407–120421.
Sergio Machado, Cláudio Portella, Julio Silva, Bruna Velasques, Patrícia Terra, Camila Ferreira-Vorkapic, Vernon Silva, Luis Miana, Luis Basile, Mauricio Cagy, Roberto Piedade, and Pedro Ribeiro. 2007. Changes in quantitative EEG absolute power during the task of catching an object in free fall. Arquivos de neuropsiquiatria 65 (10 2007), 633–6.
Christoph Mayr-Dorn, Michael Vierhauser, Stefan Bichler, Felix Keplinger, Jane Cleland-Huang, Alexander Egyed, and Thomas Mehofer. 2021. Supporting quality assurance with automated process-centric quality constraints checking. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, Madrid, ES, 1298–1310.
Júlio Medeiros, Ricardo Couceiro, Gonçalo Duarte, João Durães, João Castelhano, Catarina Duarte, Miguel Castelo-Branco, Henrique Madeira, Paulo de Carvalho, and César Teixeira. 2021. Can EEG be adopted as a neuroscience reference for assessing software programmers’ cognitive load? Sensors 21, 7 (2021), 2338.
J. Medeiros, M. Simões, J. Castelhano, R. Abreu, R. Couceiro, J. Henriques, et al. 2024. EEG as a potential ground truth for the assessment of cognitive state in software development activities: A multimodal imaging study. PLoS ONE 19, 3 (2024), e0299108.
Norman Peitek, Sven Apel, Chris Parnin, Andre Brechmann, and Janet Siegmund. 2021. Program comprehension and code complexity metrics: An fMRI study. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) (Madrid, Spain). IEEE, Madrid, Spain, 524–536.
Norman Peitek, Annabelle Bergum, Maurice Rekrut, Jonas Mucke, Matthias Nadig, Chris Parnin, Janet Siegmund, and Sven Apel. 2022. Correlates of programmer efficacy and their link to experience: a combined EEG and eye-tracking study. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore, Singapore) (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 120–131.
Norman Peitek, Janet Siegmund, Sven Apel, Christian Kästner, Chris Parnin, Anja Bethmann, Thomas Leich, Gunter Saake, and André Brechmann. 2018. A look into programmers’ heads. IEEE Transactions on Software Engineering 46, 4 (2018), 442–462.
Norman Peitek, Janet Siegmund, Chris Parnin, Sven Apel, Johannes C. Hofmeister, and André Brechmann. 2018. Simultaneous measurement of program comprehension with fMRI and eye tracking: a case study. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (Oulu, Finland) (ESEM ’18). Association for Computing Machinery, New York, NY, USA, Article 24, 10 pages.
R. Pressman. 2014. Software Engineering: A Practitioner’s Approach. McGraw-Hill Education. [link]
Patrick Rempel, Patrick Mäder, Tobias Kuschke, and Jane Cleland-Huang. 2014. Mind the gap: assessing the conformance of software traceability to relevant guidelines. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 943–954.
Saeid Sanei and J.A. Chambers. 2007. Introduction to EEG. John Wiley & Sons, Ltd, Chapter 1, 1–34.
Samuel Sanford Shapiro and Martin B Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika 52, 3-4 (1965), 591–611.
Siegel Sidney. 1957. Nonparametric statistics for the behavioral sciences. The Journal of Nervous and Mental Disease 125, 3 (1957), 497.
Janet Siegmund, André Brechmann, Sven Apel, Christian Kästner, Jörg Liebig, Thomas Leich, and Gunter Saake. 2012. Toward measuring program comprehension with functional magnetic resonance imaging. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (Cary, North Carolina) (FSE ’12). Association for Computing Machinery, New York, NY, USA, Article 24, 4 pages.
Janet Siegmund, Christian Kästner, Sven Apel, Chris Parnin, Anja Bethmann, Thomas Leich, Gunter Saake, and André Brechmann. 2014. Understanding understanding source code with functional magnetic resonance imaging. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 378–389.
Janet Siegmund, Christian Kästner, Sven Apel, André Brechmann, and Gunter Saake. 2013. Experience from measuring program comprehension - toward a general framework.
Janet Siegmund, Norman Peitek, Chris Parnin, Sven Apel, Johannes Hofmeister, Christian Kästner, Andrew Begel, Anja Bethmann, and André Brechmann. 2017. Measuring neural efficiency of program comprehension. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 140–150.
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision. IEEE Transactions on Software Engineering 50, 4 (2024), 911–936.
Barbara Weber, Thomas Fischer, and René Riedl. 2021. Brain and autonomic nervous system activity measurement in software engineering: A systematic literature review. Journal of Systems and Software 178 (2021), 110946.
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
Marvin Wyrich, Justus Bogner, and Stefan Wagner. 2023. 40 years of designing code comprehension experiments: A systematic mapping study. Comput. Surveys 56, 4 (2023), 1–42.
Martin K-C Yeh, Dan Gopstein, Yu Yan, and Yanyan Zhuang. 2017. Detecting and comparing brain activity in short program comprehension using EEG. In 2017 IEEE Frontiers in Education Conference (FIE). IEEE, Indianapolis, USA, 1–5.
Publicado
08/09/2026
Como Citar
FARIAS, Roselane Silva; TAVARES, Dhyego; MIRANDA, José Garcia Vivas; ALMEIDA, Eduardo Santana de.
Investigating Cognitive Demands in Software Testing Activities: An EEG-Based Experimental Study. In: SIMPÓSIO BRASILEIRO DE TESTES DE SOFTWARE SISTEMÁTICO E AUTOMATIZADO (SAST), 11. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 59-68.
DOI: https://doi.org/10.5753/sast.2026.31048.
