Evaluate Online: a system for large-scale assessment of reading fluency tests

• Luiz Carlos Carchedi Federal University of Juiz de Fora
• Eduardo Barrére Federal University of Juiz de Fora https://orcid.org/0000-0002-1598-5362
• Jairo Francisco de Souza Federal University of Juiz de Fora http://orcid.org/0000-0002-0911-7980

DOI: https://doi.org/10.5753/sbie.2021.217139

Abstract

Reading assessment requires specific methods, which are costly because, in most cases, they involve specialist human evaluators. To reduce the cost of large-scale assessments, it is necessary to automate parts of the process. In this paper, the Evaluate Online System is presented, which enables the automation of process steps and the reduction of the time needed to obtain results. To guarantee a low error rate, the system allows the selection of the subset with a high probability of error to undergo human verification. Our approach achieved a classification accuracy of 95.97% on a real dataset containing more than 9,000 readings.

Keywords: Learning analysis, Assessment automation, Large scale assessment, Automatic speech recognition

References

Alves, S. S. S. (2019). O papel da oralidade para o desenvolvimento do letramento em crianças da educação infantil. Seminário Interlinhas, 7(1):211–215.

Biemiller, A. (1977). Relationships between oral reading rates for letters, words, and simple text in the development of reading achievement. Reading Research Quarterly.

Black, M. P., Kazemzadeh, A., Tepperman, J., and Narayanan, S. S. (2011). Automatically assessing the abcs: Verification of children’s spoken letter-names and lettersounds. ACM Transactions on Speech and Language Processing (TSLP), 7(4):15.

Carchedi, L. C., Soares, E., Gomes Jr, J., Barrere, E., and Souza, J. (2018a). Avaliação automática da fluência em leitura para crianças em fase de alfabetização. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 29, page 11.

Carchedi, L. C., Souza, J., Barrere, E., and Mendonça, F. (2018b). Onto4la: uma ontologia para integração de dados educacionais. In Anais dos Workshops do Congresso Brasileiro de Informatica na Educação, volume 7, page 439.

Chaer, M. R. and Guimarãaes, E. d. G. A. (2012). A importância da oralidade: educação infantil e séries iniciais do ensino fundamental. Disponível em: http://pergaminho.unipam.edu.br/documents/43440/43870/a-importancia.pdf. Acesso em: 04 abril 2018.

Cucchiarini, C., Strik, H., and Boves, L. (2000). Quantitative assessment of second language learners’ fluency by means of automatic speech recognition technology. The Journal of the Acoustical Society of America, 107(2):989–999.

Deeney, T. A. (2010). One-minute fluency measures: Mixed messages in assessment and instruction. The Reading Teacher, 63(6):440–450.

Dixon, J. (2001). Evaluation tools for flexible delivery (workshop version). Melbourne: TAFE frontiers.

Gomes Jr, J., Silva, W. A., Souza, J., Barrere, E., and Souza, J. (2019). Uso de alinhadores forçados para avaliação automática em larga escala da fluência em leitura. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 30, page 61.

Gruhn, R. E., Minker, W., and Nakamura, S. (2011). Statistical pronunciation modeling for non-native speech processing. Springer Science & Business Media

Neumeyer, L., Franco, H., Weintraub, M., and Price, P. (1996). Automatic textindependent pronunciation scoring of foreign language student speech. In International Conference on Spoken Language. ICSLP 96., volume 3, pages 1457–1460. IEEE

Silva, W. A., Gomes Jr, J., Knop, I., Barrére, E., and Souza, J. (2019). Talk2me: Uma abordagem computacional para auxiliar na identificação de falhas no processo de alfabetização. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE), volume 30, page 723.

Valencia, S. W., Smith, A. T., Reece, A. M., Li, M., Wixson, K. K., and Newman, H. (2010). Oral reading fluency assessment: Issues of construct, criterion, and consequential validity. Reading Research Quarterly, 45(3):270–291.

Xie, S., Evanini, K., and Zechner, K. (2012). Exploring content features for automated speech scoring. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 103–111. Association for Computational Linguistics.

Yeung, G. and Alwan, A. (2018). On the difficulties of automatic speech recognition for kindergarten-aged children. In Proc. Interspeech 2018, pages 1661–1665.