Investigating User's Attentional Focus in Computational Environments: A Literature Review with Emphasis on Webcam Data

• Cassiano da Silva Souza PUCRS / IFSul
• Milene Selbach Silveira PUCRS
• Isabel Harb Manssour PUCRS

DOI: https://doi.org/10.5753/webmedia.2024.241929

Resumo

Maintaining the user’s attentional focus has become a recurring concern in recent years. This is due to the consolidation of remote and hybrid models for study and work, which were widely experienced during the social distancing caused by COVID-19. This paper presents a review of works that address this problem by analyzing webcam data, a promising device for behavioral studies. The literature review from 2013 to 2023 was carried out using a hybrid search strategy, through which we selected and analyzed 57 papers. The summary of this study is presented in an interactive visual survey format called the AttentionVis Browser tool. As additional contributions, we provide a list of lessons learned, a list of work limitations, and possibilities for future research.

Palavras-chave: attention monitoring, webcam, data analysis, literature review

Referências

Andrea F. Abate, Lucia Cascone, Michele Nappi, Fabio Narducci, and Ignazio Passero. 2021. Attention monitoring for synchronous distance learning. Future Generation Computer Systems 125 (2021), 774–784. DOI: 10.1016/j.future.2021.07.026

Cevat Giray Aksoy, Jose Maria Barrero, Nicholas Bloom, Steven J Davis, Mathias Dolls, and Pablo Zarate. 2023. Working from home around the globe: 2023 Report. Technical Report. EconPol Policy Brief.

Nese Alyuz, Eda Okur, Utku Genc, Sinem Aslan, Cagri Tanriover, and Asli Arslan Esme. 2017. An unobtrusive and multimodal approach for behavioral engagement detection of students. In Proceedings of the 1st International Workshop on Multimodal Interaction for Education. ACM, Glasgow, UK, 26–32. DOI: 10.1145/3139513.3139521

Sinem Aslan, Nese Alyuz, Cagri Tanriover, Sinem E. Mete, Eda Okur, Sidney K. D’Mello, and Asli Arslan Esme. 2019. Investigating the Impact of a Real-time, Multimodal Student Engagement Analytics Technology in Authentic Classrooms. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, Glasgow, Uk, 1–12. DOI: 10.1145/3290605.3300534

Yousef Atoum, Liping Chen, Alex X. Liu, Stephen D. H. Hsu, and Xiaoming Liu. 2017. Automated Online Exam Proctoring. IEEE Transactions on Multimedia 19, 7 (July 2017), 1609–1624. DOI: 10.1109/TMM.2017.2656064

Charles E. Bailey. 2007. Cognitive Accuracy and Intelligent Executive Function in the Brain and in Business. Annals of the New York Academy of Sciences 1118, 1 (2007), 122–141. DOI: 10.1196/annals.1412.011

Jose Maria Barrero, Nicholas Bloom, and Steven J Davis. 2021. Internet access and its implications for productivity, inequality, and resilience. Technical Report. National Bureau of Economic Research.

Guido Borghi, Marco Venturelli, Roberto Vezzani, and Rita Cucchiara. 2017. Poseidon: Face-from-depth for driver pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Los Alamitos, US, 5494–5503. DOI: 10.48550/arXiv.1611.10195

Benjamin T. Carter and Steven G. Luke. 2020. Best practices in eye tracking research. International Journal of Psychophysiology 155 (2020), 49–62.

Comitê Gestor da Internet no Brasil. 2022. Pesquisa sobre o uso das tecnologias de informação e comunicação nas escolas brasileiras - TIC Educação 2021 (1 ed.). Núcleo de Informação e Coordenação do Ponto BR, São Paulo, BR.

Melissa Cote, Frederic Jean, Alexandra Branzan Albu, and David Capson. 2016. Video summarization for remote invigilation of online exams. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. IEEE, Lake Placid, US, 1–9. DOI: 10.1109/WACV.2016.7477704

Adele Diamond and Kathleen Lee. 2011. Interventions Shown to Aid Executive Function Development in Children 4 to 12 Years Old. Science 333, 6045 (2011), 959–964. DOI: 10.1126/science.1204529

Fahmid Morshed Fahid, Seung Lee, Bradford Mott, Jessica Vandenberg, Halim Acosta, Thomas Brush, Krista Glazewski, Cindy Hmelo-Silver, and James Lester. 2023. Effects of Modalities in Detecting Behavioral Engagement in Collaborative Game-Based Learning. In Proceedings of the 13th International Learning Analytics and Knowledge Conference. ACM, Arlington, US, 208–218. DOI: 10.1145/3576050.3576079

Gabriele Fanelli, Thibaut Weise, Juergen Gall, and Luc Van Gool. 2011. Real Time Head Pose Estimation from Consumer Depth Cameras. Pattern Recognition 6835 (2011), 101–110. DOI: 10.1109/3DV.2014.54

Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, Yingbo Zhou, Chetan Ramaiah, Fangxiang Feng, Rui Li, Xiaojie Wang, Dimitris Athanasakis, John Shawe-Taylor, Maximilian Milakov, John Park, Radu Ionescu, Marius Popescu, Cristian Grozea, James Bergstra, Jingjing Xie, Lukasz Romaszko, Bing Xu, Zhang Chuang, and Yoshua Bengio. 2015. Challenges in representation learning: A report on three machine learning contests. Neural Networks 64 (2015), 59–63. DOI: 10.1016/j.neunet.2014.09.005

Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. 2017. CNN architectures for large-scale audio classification. In Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, IEEE Press, New Orleans, US, 131–135. DOI: 10.48550/arXiv.1609.09430

Basavaraj N Hiremath, Anushree Mitra, Aman Thapa, S Amoolya, and A Tameem. 2023. Proctoring using Artificial Intelligence. SSRN Electronic Journal (2023). DOI: 10.2139/ssrn.4411332

Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. 2007. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments. Technical Report 07-49. University of Massachusetts, Amherst.

Instituto Semesp. 2023. Mapa do Ensino Superior no Brasil. Semesp, São Paulo, BR. 13a ed. [link]

Mohamed Irfan, Mohammed Aslam, Ziyan Maraikar, Upul Jayasinghe, and Mohamed Fawzan. 2021. Ensuring Academic Integrity of Online Examinations. In Proceedings of the IEEE 16th International Conference on Industrial and Information Systems. IEEE, Kandy, LK, 295–300. DOI: 10.1109/ICIIS53135.2021.9660737

Amr Jadi. 2021. New Detection Cheating Method of Online-Exams during COVID-19 Pandemic. International Journal of Computer Science and Network Security 21, 4 (April 2021), 123–130. DOI: 10.22937/IJCSNS.2021.21.4.17

Marcos Kalinowski, Tatiana Escovedo, Hugo Villamizar, and Hélio Lopes. 2023. Engenharia de Software para Ciência de Dados: Um guia de boas práticas com ênfase na construção de sistemas de Machine Learning em Python. Casa do Código, São Paulo, BR.

Virgínia Kastrup. 2007. O funcionamento da atenção no trabalho do cartógrafo. Psicologia e Sociedade 19, 1 (Jan 2007), 15–22. DOI: 10.1590/S0102-71822007000100003

Amanjot Kaur, Aamir Mustafa, Love Mehta, and Abhinav Dhall. 2018. Prediction and Localization of Student Engagement in the Wild. In Proceedings of the Digital Image Computing: Techniques and Applications. IEEE, Canberra, AU, 1–8. DOI: 10.48550/arXiv.1804.00858

S. Keshav. 2007. How to Read a Paper. Special Interest Group on Data Communication da Association for Computing Machinery 37, 3 (jul 2007), 83–84. DOI: 10.1145/1273445.1273458

Ahsan Raza Khan, Sara Khosravi, Sajjad Hussain, Rami Ghannam, Ahmed Zoha, and Muhammad Ali Imran. 2022. EXECUTE: Exploring Eye Tracking to Support E-learning. In Proceedings of the IEEE Global Engineering Education Conference. IEEE, Tunis, TN, 670–676. DOI: 10.1109/EDUCON52537.2022.9766506

Barbara Kitchenham and Stuart Charters. 2007. Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-001. University of Durham.

Kostiantyn Kucher and Andreas Kerren. 2015. Text visualization techniques: Taxonomy, visual survey, and community insights. In Proceedings of the IEEE Pacific Visualization Symposium. IEEE, Hangzhou, CN, 117–121. DOI: 10.1109/PACIFICVIS.2015.7156366

Taeckyung Lee, Dain Kim, Sooyoung Park, Dongwhi Kim, and Sung-Ju Lee. 2022. Predicting Mind-Wandering with Facial Videos in Online Lectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. IEEE, New Orleans, US, 2103–2112. DOI: 10.1109/CVPRW56347.2022.00228

Haotian Li, Min Xu, Yong Wang, Huan Wei, and Huamin Qu. 2021. A visual analytics approach to facilitate the proctoring of online exams. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, New York, US, 1–17. DOI: 10.1145/3411764.3445294

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Proceedings of the 2014 Conference on Computer Vision, Vol. 8693. Springer, Zurich, CH, 740–755. DOI: 10.48550/arXiv.1405.0312

Jens Madsen, Sara U. Júlio, Pawel J. Gucik, Richard Steinberg, and Lucas C. Parra. 2021. Synchronized eye movements predict test scores in online video education. Proceedings of the National Academy of Sciences 118, 5 (2021). DOI: 10.1073/pnas.2016980118

Rachna Maithani. 2023. Securelens: Enhancing Trustworthiness in Remote Exams with Opencv Protective Eye. Journal of Modernization in Engineering Technology and Science 05, 6 (2023), 1857–1861.

Margaret W Matlin. 2004. Psicologia cognitiva, 5a ed. LTC, Rio de Janeiro, BR.

Samuel Monteiro, Rutuja Bhate, Lav Sharma, and Phiroj Shaikh. 2022. Proct-Xam – AI Based Proctoring. In Proceedings of the 2nd Asian Conference on Innovation in Technology. IEEE, Ravet, IN, 1–6. DOI: 10.1109/ASIANCON55314.2022.9908817

Erica Mourão, Marcos Kalinowski, Leonardo Murta, Emilia Mendes, and Claes Wohlin. 2017. Investigating the Use of a Hybrid Search Strategy for Systematic Reviews. In Proceedings of the 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, Toronto, CA, 193–198. DOI: 10.1109/ESEM.2017.30

Makoto Nakayama and Charlie C Chen. 2022. Digital Distractions and Remote Work: A Balancing Act at Home. Information Resources Management Journal 35, 1 (2022), 1–17. DOI: 10.4018/IRMJ.308675

Azmi Can Ozgen, Mahiye Uluyagmur Ozturk, and Umut Bayraktar. 2021. Cheating Detection Pipeline for Online Interviews and Exams. ArXiv abs/2106.14483 (2021), 4. DOI: 10.48550/arXiv.2106.14483

Azmi Can Ozgen, Mahiye Uluyagmur Ozturk, Orkun Torun, Jianguo Yang, and Mehmet Zahit Alparslan. 2021. Cheating Detection Pipeline for Online Interviews. In Proceedings of the 29th Signal Processing and Communications Applications Conference. IEEE, Istanbul, TR, 1–4. DOI: 10.1109/SIU53274.2021.9477950

Kim Parker. 2023. About a third of U.S. workers who can work from home now do so all the time. [link] Acessado: 07-02-2024.

Leandro Persona, Fernando Meloni, and Alessandra Alaniz Macedo. 2023. An accurate real-time method to detect the smile facial expression. In Proceedings of the 29th Brazilian Symposium on Multimedia and the Web (Ribeirão Preto, Brazil) (WebMedia ’23). Association for Computing Machinery, New York, USA, 46–55. DOI: 10.1145/3617023.3617031

Michael I Posner. 1980. Orienting of attention. Quarterly journal of experimental psychology 32, 1 (1980), 3–25. DOI: 10.1080/00335558008248231

Tejaswi Potluri, Venkatarama Phani Kumar, and K Venkata Krishna Kishore. 2023. An automated online proctoring system using attentive-net to assess student mischievous behavior. Multimedia Tools and Applications 82, 20 (Aug. 2023), 30375–30404. DOI: 10.1007/s11042-023-14604-w

Tejaswi Potluri, Venkatarama Phani Kumar Sistla, and Drkv Krishna. 2023. Detecting autism of examinee in automated online proctoring using eye-tracking. Journal of Theoretical and Applied Information Technology 101, 3 (2023).

Swathi Prathish, Athi Narayanan S., and Kamal Bijlani. 2016. An intelligent system for online exam monitoring. In Proceedings of the International Conference on Information Science. IEEE, Kochi, IN, 138–143. DOI: 10.1109/INFOSCI.2016.7845315

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision 115, 3 (2015), 211–252. DOI: 10.1007/s11263-015-0816-y

Andrew Sanders, Bradley Boswell, Andrew Allen, Gursimran Singh Walia, and Md Shakil Hossain. 2022. Development and Field-Testing of a Non-intrusive Classroom Attention Tracking System (NiCATS) for Tracking Student Attention in CS Classrooms. In Proceedings of the IEEE Frontiers in Education Conference. IEEE, Uppsala, SE, 1–9. DOI: 10.1109/FIE56618.2022.9962447

Andrew Sanders, Bradley Boswell, Gursimran Singh Walia, and Andrew Allen. 2021. Non-Intrusive Classroom Attention Tracking System (NiCATS). In Proceedings of the IEEE Frontiers in Education Conference. IEEE, Lincoln, US, 1–9. DOI: 10.1109/FIE49875.2021.9637411

Steven Seiden, Long Huang, and Chen Wang. 2023. Poster: Snooping Online Form Choice Privacy in Video Calls. In Proceedings of the IEEE Symposium on Security and Privacy. IEEE, San Francisco, US, 1–2.

Abdelrahman Shata, Zineddine N. Haitaamar, and Abdsamad Benkrid. 2023. Baseline Estimation in Face Detection for AI Proctored Examinations through Convoluted Neural Networks. In Proceedings of the International Conference on IT Innovation and Knowledge Discovery. IEEE, Manama, BH, 1–5. DOI: 10.1109/ITIKD56332.2023.10100328

Merzoug Soltane and Mohamed Ridda Laouar. 2021. A Smart System to Detect Cheating in the Online Exam. In Proceedings of the International Conference on Information Systems and Advanced Technologies. IEEE, Tebessa, DZ, 1–5. DOI: 10.1109/ICISAT54145.2021.9678418

Robert J Sternberg. 2008. Psicologia cognitiva (4a ed.). artmed, Rio de Janeiro, BR.

P. Tejaswi, S. Venkatramaphanikumar, and K. Venkata Krishna Kishore. 2023. Proctor net: An AI framework for suspicious activity detection in online proctored examinations. Measurement 206 (Jan. 2023), 112266. DOI: 10.1016/j.measurement.2022.112266

Shogo Terai, Shizuka Shirai, Mehrasa Alizadeh, Ryosuke Kawamura, Noriko Takemura, Yuki Uranishi, Haruo Takemura, and Hajime Nagahara. 2020. Detecting Learner Drowsiness Based on Facial Expressions and Head Movements in Online Courses. In Proceedings of the 25th International Conference on Intelligent User Interfaces Companion. ACM, Cagliari, IT, 124–125. DOI: 10.1145/3379336.3381500

Jian-Wei Tzeng, Cheng-Yu Hsueh, Chia-An Lee, and Wei-Yun Shih. 2023. Identifying the Correlation Between Online Exam Answer Trajectory and Test Behavior Based on Artificial Intelligence and Eye Movement Detection Technology. In Proceedings of the International Conference on Consumer Electronics. IEEE, PingTung, TW, 503–504. DOI: 10.1109/ICCE-Taiwan58799.2023.10226745

Puru Verma, Neil Malhotra, Ram Suri, and Rajesh Kumar. 2024. Automated smart artificial intelligence-based proctoring system using deep learning. Soft Computing 28, 4 (Feb. 2024), 3479–3489. DOI: 10.1007/s00500-023-08696-7

Chenghao Wang et al. 2022. Comprehensively summarizing what distracts students from online learning: A literature review. Human Behavior and Emerging Technologies 2022, 1 (2022), 1483531. DOI: 10.1155/2022/1483531

Jacob Whitehill, Zewelanji Serpell, Yi-Ching Lin, Aysha Foster, and Javier R. Movellan. 2014. The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions. IEEE Transactions on Affective Computing 5, 1 (Jan. 2014), 86–98. DOI: 10.1109/TAFFC.2014.2316163

Claes Wohlin. 2014. Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (London, England, United Kingdom) (EASE ’14). Association for Computing Machinery, New York, US, 10 pages. DOI: 10.1145/2601248.2601268

Claes Wohlin, Marcos Kalinowski, Katia Romero Felizardo, and Emilia Mendes. 2022. Successful combination of database search and snowballing for identification of primary studies in systematic literature studies. Information and Software Technology 147 (2022), 106908. DOI: 10.1016/j.infsof.2022.106908

Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. 2014. Learning face representation from scratch. arXiv preprint 1411.7923 (2014), 9 pages. DOI: 10.48550/arXiv.1411.7923

Intan Nurma Yulita, Fauzan Akmal Hariz, Ino Suryana, and Anton Satria Prabuwono. 2023. Educational Innovation Faced with COVID-19: Deep Learning for Online Exam Cheating Detection. Education Sciences 13, 2 (Feb. 2023), 194. DOI: 10.3390/educsci13020194

Cheng Zhang, Cheng Chang, Lei Chen, and Yang Liu. 2018. Online Privacy-Safe Engagement Tracking System. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. ACM, Boulder, US, 553–554. DOI: 10.1145/3242969.3266295

Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z Li. 2016. Face alignment across large poses: A 3D solution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Las Vegas, US, 146–155. DOI: 10.1109/CVPR.2016.23