Assessment of reading fluency in Portuguese: first experience with large-scale use of AI for assessment in Brazil

• Caio C. Rocha Federal University of Juiz de Fora
• Rômulo C. de Mello Federal University of Juiz de Fora https://orcid.org/0000-0001-5023-6183
• Jairo F. de Souza Federal University of Juiz de Fora https://orcid.org/0000-0002-0911-7980

DOI: https://doi.org/10.5753/sbie.2024.244971

Abstract

With recent advances in AI, the use of automatic systems for assessing learning has grown. This study presents the results of the first large-scale automatic assessment of reading fluency in Portuguese carried out with second-year elementary school students. The results were compared with a significant volume of manual corrections performed by trained correctors. Although the solution doesn't cover all the data produced by humans, the results show that automatic marking is compatible with the data produced by humans, constituting a viable and economical alternative for large-scale assessments.

Keywords: Large-scale assessment, AI in Education, Reading fluency

References

Almeida Silva, W., Carchedi, L., Gomes Jr, J., Souza, J., Barrere, E., and Souza, J. (2021). A framework for large-scale automatic fluency assessment. International Journal of Distance Education Technologies, 19.

Bailly, G., Godde, E., Piat-Marchand, A.-L., and Bosse, M.-L. (2022). Automatic assessment of oral readings of young pupils. Speech Communication, 138:67–79.

Bernstein, J. C. and Cheng, J. (2023). Speech analysis in assessment. Advancing Natural Language Processing in Educational Assessment, pages 31–57.

Bolaños, D., Cole, R., Ward, W., Tindal, G., Hasbrouck, J., and Schwanenflugel, P. (2013). Human and automated assessment of oral reading fluency. Journal of Educational Psychology, 105:1142.

Coscarelli, C. V. (2002). Entendendo a leitura. Revista de estudos da linguagem, 10(1):7–27.

de Assis, E., Ferreira, A. L., Silva, C., and de Souza, J. (2022). Classificação automática de áudios de leituras de pseudopalavras para avaliação em larga escala de fluência da leitura de crianças em fase de alfabetização. In Anais do XXXIII Simpósio Brasileiro de Informática na Educação, pages 27–38, Porto Alegre, RS, Brasil. SBC.

Forero-Corba, W. and Bennasar, F. N. (2024). Techniques and applications of machine learning and artificial intelligence in education: a systematic review. RIED-Revista Iberoamericana de Educación a Distancia, 27(1).

González-Calatayud, V., Prendes-Espinosa, P., and Roig-Vila, R. (2021). Artificial intelligence for student assessment: A systematic review. Applied Sciences, 11(12):5467.

Grosman, J. (2022). Fine-tuned XLS-R 1B model for speech recognition in Portuguese. [link].

Jones, C. (2005). Teachers need help too: aiding the marking process through a human-computer collaborative approach. In Human Centred Technology Workshop 2005, page 56.

Lemle, M. (1987). Guia teórico do alfabetizador. Ed. Ática.

Memarian, B. and Doleck, T. (2023). A review of assessment for learning with artificial intelligence. Computers in Human Behavior: Artificial Humans, page 100040.

Messer, M., Brown, N. C., Kölling, M., and Shi, M. (2024). Automated grading and feedback tools for programming education: A systematic review. ACM Transactions on Computing Education, 24(1):1–43.

Pinheiro, A. and Vilhena, D. (2022). Teste de reconhecimento de palavras e pseudopalavras: validades de conteúdo e externa. Signo, 47:147–164.

Proença, J., Lopes, C., Tjalve, M., Stolcke, A., Candeias, S., and Perdigão, F. (2017). Automatic evaluation of children reading aloud on sentences and pseudowords. In INTERSPEECH, pages 2749–2753.

Rodrigues, A., Ribeiro, G., Silva, V., Carvalho, W., Ramírez, M., Alves, L., Finger, M., Navas, A. L., and Ribeiro, C. (2023). Ai and reading fluency for brazilian portuguese: A preliminary study. Preprint available at SSRN 4429229.

Silva, C. N., Ferreira, A. L. V., de Assis, E. C., and de Souza, J. F. (2022). Definição de heurística para identificação automática da fluência em leitura de crianças em fase de alfabetização. In Anais do XXXIII Simpósio Brasileiro de Informática na Educação, pages 39–50. SBC.

Soares, E., Carchedi, L. C., Gomes Jr, J., Barrére, E., and Souza, J. (2018). Avaliação automática da fluência em leitura para crianças em fase de alfabetizaçãoo. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação), volume 29, page 11.

Xu, L. (2020). The dilemma and countermeasures of ai in educational application. In Proceedings of the 2020 4th International Conference on Computer Science and Artificial Intelligence, pages 289–294.

Yildiz, M., Keskin, H. K., Oyucu, S., Hartman, D. K., Temur, M., and Aydogmus, M. (2024). Can artificial intelligence identify reading fluency and level? comparison of human and machine performance. Reading & Writing Quarterly, pages 1–18.

Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., Liu, J.-B., Yuan, J., and Li, Y. (2021). A review of artificial intelligence (ai) in education from 2010 to 2020. omplexity, 2021(1):8812542.

Zhang, J., Pan, P., and Yan, Y. (2012). Automatic scoring on english passage reading quality. Procedia Engineering, 29:2744–2748. 2012 International Workshop on Information and Electronics Engineering.