Assessing the Development of Critical Thinking in Programming Education for Technical and Higher Education Students: A Proposed Model
Abstract
This paper proposes the development and validation of a model to assess critical thinking in programming education for technical and higher education students. Using the Evidence-Centered Design (ECD) methodology and the Principles of Investigation-Based Assessment (PADI) conceptions, the model was developed to capture skills such as interpretation, analysis, and evaluation. The Content and Construct Validity were evaluated by a panel of experts, showing promising results, such as the model's ability to accurately measure the intended aspects. Thus, the model demonstrates significant potential to be an effective tool in computer education.
References
ACM/IEEE-CS Joint Task Force on Computing Curricula (2023) Computer Science Curricula 2023. Technical report. ACM Press and IEEE Computer Society Press. Disponível em: [link] [Acesso em: 12 março 2025].
Alexandre, N. M. C. & Coluci, M. Z. O. (2011) “Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas”, Ciência & Saúde Coletiva, 16(7), pp. 3061-3068. .
Araújo, L., Pessoa, M., & Pires, F. (2024b). Investigando a relação entre pensamento computacional e narrativas digitais e não digitais. In Anais do 35º Simpósio Brasileiro de Informática na Educação (SBIE) (pp. 3294-3303). Porto Alegre, RS: Sociedade Brasileira de Computação. DOI: 10.5753/sbie.2024.244972.
Arndt, D. M., Martins, R. M., & Hauck, J. C. R. (2024). Critical thinking assessment in K-12 computing education: A systematic mapping. Informatics in Education. DOI: 10.15388/infedu.2025.02
Arndt, D. M., Martins, R. M., & Hauck, J. C. R. (2024). Avaliação de Habilidades do Pensamento Crítico no Ensino Técnico e Superior no Ensino de Computação: Um Mapeamento Sistemático. Revista Brasileira de Informática na Educação (RBIE) (aceito).
Beecham, S., Hall, T., Britton, C., Cottee, M. & Rainer, A. (2005) “Using an expert panel to validate a requirements process improvement model”, Journal of Systems and Software, 76(3), pp. 251-275. .
Catojo, A. R. S., & Nunes, M. A. S. N. (2024). O pensamento computacional para o desenvolvimento de aprendizagens de leitura e pensamento críticos no ensino fundamental: Um mapeamento sistemático da literatura. Revista Brasileira de Informática na Educação, 32(1), 135-156. DOI: 10.1016/j.tsc.2017.05.005.
Cicchetti, D. V., & Sparrow, S. A. (1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86(2), 127-137.
Critical Thinking (2019) “Critical thinking testing and assessment”. Foundation for Critical Thinking. Disponível em: [link] [Acesso em: 12 março 2025].
Davis, L. L. (1992) “Instrument review: Getting the most from a panel of experts”, Applied Nursing Research, 5(4), pp. 194-197. .
Derksen, B. M., Bruinsma, W., Goslings, J.C. & Schep, N. W.L. (2024). The Kappa Paradox Explained. The Journal of Hand Surgery, 49(5), pp. 482-485. DOI: 10.1016/j.jhsa.2024.01.006. .
Facione, P. A., & Facione, N. C. (1990). Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction. ERIC, Institute of Education Sciences.
Gong, D., Yang, H. H., & Cai, J. (2020). Exploring the key influencing factors on college students’ computational thinking skills through flipped-classroom instruction. International Journal of Educational Technology in Higher Education, 17(19). DOI: 10.1186/s41239-020-00196-0. .
Graham, M. & Milanowski, A. & Westat, J. (2014). Measuring and Promoting Inter-Rater Agreement of Teacher and Principal Performance Ratings. .
Grant, J. S. & Davis, L. L. (1997) “Selection and use of content experts for instrument development”, Research in Nursing & Health, 20(3), pp. 269-274. .
Haynes, S. N., Richard, D. C. S. & Kubany, E. S. (1995) “Content validity in psychological assessment: A functional approach to concepts and methods”, Psychological Assessment, 7(3), pp. 238-247. .
Huang, X. & Qiao, C. (2024) “Enhancing computational thinking skills through artificial intelligence education at a STEAM high school”, Science & Education, 33. DOI: 10.1007/s11191-022-00392-6. [Acesso em: 12 março 2025]. .
Insight Assessment (2023) “Critical thinking assessments for higher education”. Disponível em: [link]. [Acesso em: 12 março 2025].
Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (2023) "Índice de Desenvolvimento da Educação Básica: Ideb 2023", Disponível em: [link]. [Acesso em: 12 março 2025]
Instituto Ayrton Senna, s.d. Criatividade e pensamento crítico. Disponível em: [link] [Acesso em: 12 março 2025].
Kaur, A., & Chahal, K. K. (2023). Exploring personality and learning motivation influences on students’ computational thinking skills in introductory programming courses. Journal of Science Education and Technology, 32(5), pp. 778–792. DOI: 10.1007/s10956-023-10052-1. .
K–12 Computer Science Framework (2023) “K–12 Computer Science Framework”, Disponível em: [link] [Acesso em: 12 março 2025].
Kendall, M.G. (1948). Rank correlation methods. Griffin. .
Landis, J. R. & Koch, G. G. (1977) “The measurement of observer agreement for categorical data”, Biometrics, 33(1), pp. 159-174. .
Lawshe, C. H. (1975) “A quantitative approach to content validity”, Personnel Psychology, 28(4), pp. 563-575. .
Lee, S., Choi, D., Lee, M., Choi, J. & Lee, S. (2023) “Fostering youth's critical thinking competency about AI through exhibition”, in CHI Conference on Human Factors in Computing Systems, Hamburg, Germany. .
Lin, P.-H. & Chen, S.-Y. (2020) “Design and Evaluation of a Deep Learning Recommendation Based Augmented Reality System for Teaching Programming and Computational Thinking”, In: IEEE Access, vol. 8, pp. 45689-45699, DOI: 10.1109/ACCESS.2020.2977679. .
Liu, J. L., McBride, R. E., Xiang, P., & Scarmardo-Rhodes, M. (2018). Physical education pre-service teachers’ understanding, application, and development of critical thinking. Quest, 70(1), pp. 23-38. DOI: 10.1080/00336297.2017.1325205. .
Lynn, M. (1986) “Determination and Quantification of Content Validity Index”, Nursing Research, 35, pp. 382-386. DOI: 10.1097/00006199-198611000-00017. .
Mäkiö, E. & Mäkiö, J. (2023) "The task-based approach to teaching critical thinking for computer science students", Education Sciences, 13(7), pp. 742. DOI: 10.3390/educsci13070742. .
Messick, S. (1995) “Validity of psychological assessment”, American Psychologist, 50(9), pp. 741-749. .
Ministério da Educação (MEC) (2017) "Base Nacional Comum Curricular", Disponível em: [link].
Mislevy, R. J., Almond, R. G. & Lukas, J. F. (2003) “A Brief Introduction to Evidence-Centered Design”, ETS Research Report Series, i–29. .
Moskal, B. M.; Leydens, J. A. (2000) Scoring rubric development: Validity and reliability. Practical Assessment, Research & Evaluation, v. 7, n. 10. Disponível em: [link]. [Acesso em: 12 março 2025]. .
OECD. (2019) OECD Learning Compass 2030: “A series of concept notes. OECD Learning Compass 2030 Concept Note Series”. Disponível em: [link]. [Acesso em: 12 março 2025].
OECD (2024) “Teaching, Learning and Assessing Creative and Critical Thinking Skills”, Disponível em: [link]. [Acesso em: 12 março 2025].
Paul, J. A.; Sinha, M.; Cochran, J. D. (2023) “Instruments to assess students critical thinking—A qualitative approach. Decision Sciences Journal of Innovative Education, v. 21, n. 3. .
Riconcente, M. M., Mislevy. J. & Hamel, L. (2005). “An introduction to PADI task templates, Menlo Park: SRI International. (PADI Technical Report 3)”. Disponívelem: [link]. [Acesso em: 10 janeiro 2025].
Sari, M.K., Sudiyanto, S. & Kurniawan, S.B. (2022) “Critical thinking skills profile of fourth-grade elementary school students in science learning”, Proceedings of the 5th International Conference on Learning Innovation and Quality Education, Surakarta, Indonesia. .
Seeratan, K. L. & Mislevy, R. J. (2008) “Design patterns for assessing internal knowledge representations”, SRI International, Menlo Park. (PADI Technical Report 22). Disponível em: [link]. [Acesso em: 10 janeiro 2025].
Sociedade Brasileira de Computação-SBC (2021) “Educação superior em computação: Estatísticas – 2021”. Available at: [link]. [Acesso em: 12 março 2025].
Song, D., Hong, H., & Oh, E. Y. (2021). Applying computational analysis of novice learners' computer programming patterns to reveal self-regulated learning, computational thinking, and learning performance. Computers in Human Behavior, 120, 106746. DOI: 10.1016/j.chb.2021.106746. .
SPSS (2025) Kendall's W Test in SPSS Disponível em: [link]. [Acesso em: 12 março 2025].
Taylor, W. (2022) “Promoting critical thinking through classroom discussion”, in FUIKS, C. L. and CLARK, L. (Eds.) Teaching and learning in honors. .
UNICEF. Artificial intelligence chatbots. (2023). Disponível em: [link]. [Acesso em: 25 fevereiro 2025].
World Economic Forum (2020) “The Future of Jobs Report 2020”. Available at: [link] [Acesso em: 12 março 2025].
Yeh, Y. C. (2003) Critical thinking test-level I (CTT-I). Taipei: Psychological Publishing.
Zawacki-Richter, O., Marín, V. I., Bond, M. and Gouverneur, F. (2019) “Systematic review of research on artificial intelligence applications in higher education—where are the educators?”, International Journal of Educational Technology in Higher Education, 16(39). DOI: 10.1186/s41239-019-0172-8. .
Alexandre, N. M. C. & Coluci, M. Z. O. (2011) “Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas”, Ciência & Saúde Coletiva, 16(7), pp. 3061-3068. .
Araújo, L., Pessoa, M., & Pires, F. (2024b). Investigando a relação entre pensamento computacional e narrativas digitais e não digitais. In Anais do 35º Simpósio Brasileiro de Informática na Educação (SBIE) (pp. 3294-3303). Porto Alegre, RS: Sociedade Brasileira de Computação. DOI: 10.5753/sbie.2024.244972.
Arndt, D. M., Martins, R. M., & Hauck, J. C. R. (2024). Critical thinking assessment in K-12 computing education: A systematic mapping. Informatics in Education. DOI: 10.15388/infedu.2025.02
Arndt, D. M., Martins, R. M., & Hauck, J. C. R. (2024). Avaliação de Habilidades do Pensamento Crítico no Ensino Técnico e Superior no Ensino de Computação: Um Mapeamento Sistemático. Revista Brasileira de Informática na Educação (RBIE) (aceito).
Beecham, S., Hall, T., Britton, C., Cottee, M. & Rainer, A. (2005) “Using an expert panel to validate a requirements process improvement model”, Journal of Systems and Software, 76(3), pp. 251-275. .
Catojo, A. R. S., & Nunes, M. A. S. N. (2024). O pensamento computacional para o desenvolvimento de aprendizagens de leitura e pensamento críticos no ensino fundamental: Um mapeamento sistemático da literatura. Revista Brasileira de Informática na Educação, 32(1), 135-156. DOI: 10.1016/j.tsc.2017.05.005.
Cicchetti, D. V., & Sparrow, S. A. (1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86(2), 127-137.
Critical Thinking (2019) “Critical thinking testing and assessment”. Foundation for Critical Thinking. Disponível em: [link] [Acesso em: 12 março 2025].
Davis, L. L. (1992) “Instrument review: Getting the most from a panel of experts”, Applied Nursing Research, 5(4), pp. 194-197. .
Derksen, B. M., Bruinsma, W., Goslings, J.C. & Schep, N. W.L. (2024). The Kappa Paradox Explained. The Journal of Hand Surgery, 49(5), pp. 482-485. DOI: 10.1016/j.jhsa.2024.01.006. .
Facione, P. A., & Facione, N. C. (1990). Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction. ERIC, Institute of Education Sciences.
Gong, D., Yang, H. H., & Cai, J. (2020). Exploring the key influencing factors on college students’ computational thinking skills through flipped-classroom instruction. International Journal of Educational Technology in Higher Education, 17(19). DOI: 10.1186/s41239-020-00196-0. .
Graham, M. & Milanowski, A. & Westat, J. (2014). Measuring and Promoting Inter-Rater Agreement of Teacher and Principal Performance Ratings. .
Grant, J. S. & Davis, L. L. (1997) “Selection and use of content experts for instrument development”, Research in Nursing & Health, 20(3), pp. 269-274. .
Haynes, S. N., Richard, D. C. S. & Kubany, E. S. (1995) “Content validity in psychological assessment: A functional approach to concepts and methods”, Psychological Assessment, 7(3), pp. 238-247. .
Huang, X. & Qiao, C. (2024) “Enhancing computational thinking skills through artificial intelligence education at a STEAM high school”, Science & Education, 33. DOI: 10.1007/s11191-022-00392-6. [Acesso em: 12 março 2025]. .
Insight Assessment (2023) “Critical thinking assessments for higher education”. Disponível em: [link]. [Acesso em: 12 março 2025].
Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (2023) "Índice de Desenvolvimento da Educação Básica: Ideb 2023", Disponível em: [link]. [Acesso em: 12 março 2025]
Instituto Ayrton Senna, s.d. Criatividade e pensamento crítico. Disponível em: [link] [Acesso em: 12 março 2025].
Kaur, A., & Chahal, K. K. (2023). Exploring personality and learning motivation influences on students’ computational thinking skills in introductory programming courses. Journal of Science Education and Technology, 32(5), pp. 778–792. DOI: 10.1007/s10956-023-10052-1. .
K–12 Computer Science Framework (2023) “K–12 Computer Science Framework”, Disponível em: [link] [Acesso em: 12 março 2025].
Kendall, M.G. (1948). Rank correlation methods. Griffin. .
Landis, J. R. & Koch, G. G. (1977) “The measurement of observer agreement for categorical data”, Biometrics, 33(1), pp. 159-174. .
Lawshe, C. H. (1975) “A quantitative approach to content validity”, Personnel Psychology, 28(4), pp. 563-575. .
Lee, S., Choi, D., Lee, M., Choi, J. & Lee, S. (2023) “Fostering youth's critical thinking competency about AI through exhibition”, in CHI Conference on Human Factors in Computing Systems, Hamburg, Germany. .
Lin, P.-H. & Chen, S.-Y. (2020) “Design and Evaluation of a Deep Learning Recommendation Based Augmented Reality System for Teaching Programming and Computational Thinking”, In: IEEE Access, vol. 8, pp. 45689-45699, DOI: 10.1109/ACCESS.2020.2977679. .
Liu, J. L., McBride, R. E., Xiang, P., & Scarmardo-Rhodes, M. (2018). Physical education pre-service teachers’ understanding, application, and development of critical thinking. Quest, 70(1), pp. 23-38. DOI: 10.1080/00336297.2017.1325205. .
Lynn, M. (1986) “Determination and Quantification of Content Validity Index”, Nursing Research, 35, pp. 382-386. DOI: 10.1097/00006199-198611000-00017. .
Mäkiö, E. & Mäkiö, J. (2023) "The task-based approach to teaching critical thinking for computer science students", Education Sciences, 13(7), pp. 742. DOI: 10.3390/educsci13070742. .
Messick, S. (1995) “Validity of psychological assessment”, American Psychologist, 50(9), pp. 741-749. .
Ministério da Educação (MEC) (2017) "Base Nacional Comum Curricular", Disponível em: [link].
Mislevy, R. J., Almond, R. G. & Lukas, J. F. (2003) “A Brief Introduction to Evidence-Centered Design”, ETS Research Report Series, i–29. .
Moskal, B. M.; Leydens, J. A. (2000) Scoring rubric development: Validity and reliability. Practical Assessment, Research & Evaluation, v. 7, n. 10. Disponível em: [link]. [Acesso em: 12 março 2025]. .
OECD. (2019) OECD Learning Compass 2030: “A series of concept notes. OECD Learning Compass 2030 Concept Note Series”. Disponível em: [link]. [Acesso em: 12 março 2025].
OECD (2024) “Teaching, Learning and Assessing Creative and Critical Thinking Skills”, Disponível em: [link]. [Acesso em: 12 março 2025].
Paul, J. A.; Sinha, M.; Cochran, J. D. (2023) “Instruments to assess students critical thinking—A qualitative approach. Decision Sciences Journal of Innovative Education, v. 21, n. 3. .
Riconcente, M. M., Mislevy. J. & Hamel, L. (2005). “An introduction to PADI task templates, Menlo Park: SRI International. (PADI Technical Report 3)”. Disponívelem: [link]. [Acesso em: 10 janeiro 2025].
Sari, M.K., Sudiyanto, S. & Kurniawan, S.B. (2022) “Critical thinking skills profile of fourth-grade elementary school students in science learning”, Proceedings of the 5th International Conference on Learning Innovation and Quality Education, Surakarta, Indonesia. .
Seeratan, K. L. & Mislevy, R. J. (2008) “Design patterns for assessing internal knowledge representations”, SRI International, Menlo Park. (PADI Technical Report 22). Disponível em: [link]. [Acesso em: 10 janeiro 2025].
Sociedade Brasileira de Computação-SBC (2021) “Educação superior em computação: Estatísticas – 2021”. Available at: [link]. [Acesso em: 12 março 2025].
Song, D., Hong, H., & Oh, E. Y. (2021). Applying computational analysis of novice learners' computer programming patterns to reveal self-regulated learning, computational thinking, and learning performance. Computers in Human Behavior, 120, 106746. DOI: 10.1016/j.chb.2021.106746. .
SPSS (2025) Kendall's W Test in SPSS Disponível em: [link]. [Acesso em: 12 março 2025].
Taylor, W. (2022) “Promoting critical thinking through classroom discussion”, in FUIKS, C. L. and CLARK, L. (Eds.) Teaching and learning in honors. .
UNICEF. Artificial intelligence chatbots. (2023). Disponível em: [link]. [Acesso em: 25 fevereiro 2025].
World Economic Forum (2020) “The Future of Jobs Report 2020”. Available at: [link] [Acesso em: 12 março 2025].
Yeh, Y. C. (2003) Critical thinking test-level I (CTT-I). Taipei: Psychological Publishing.
Zawacki-Richter, O., Marín, V. I., Bond, M. and Gouverneur, F. (2019) “Systematic review of research on artificial intelligence applications in higher education—where are the educators?”, International Journal of Educational Technology in Higher Education, 16(39). DOI: 10.1186/s41239-019-0172-8. .
Published
2025-07-20
How to Cite
ARNDT, Deise Monquelate; MARTINS, Ramon Mayor; HAUCK, Jean C. R..
Assessing the Development of Critical Thinking in Programming Education for Technical and Higher Education Students: A Proposed Model. In: WORKSHOP ON COMPUTING EDUCATION (WEI), 33. , 2025, Maceió/AL.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 264-277.
ISSN 2595-6175.
DOI: https://doi.org/10.5753/wei.2025.7944.
