Enriquecimento de Embeddings Multimodais com Representações de Encoders de Fala para Pontuação Automática de Fluência de Leitura
Resumo
A avaliação automática da fluência de leitura é um problema relevante para o apoio a contextos educacionais, mas ainda há uma lacuna na literatura quanto a comparações sistemáticas entre representações pré-treinadas, de uma discussão clara sobre a formulação adequada da tarefa e de análises críticas sobre vieses demográficos. Neste trabalho, avaliamos quatro grupos de embeddings multimodais (Wav2Vec2, Whisper, BERT e Gemini Embedding 2, isolados ou combinados) e oito regressores ensemble no dataset SpeechOcean762 para realizar um benchmark de fluência. Os resultados mostram que a configuração Whisper + Gemini + PCA-256 com Stacking/Voting alcança melhores resultados com PCC de 0,77 e Kappa de Cohen de 0,76. Além disso, foi realizado um estudo de viés demográfico por gênero e faixa etária para avaliar possíveis viéses nos modelos.
Palavras-chave:
Fluência de leitura, Embeddings multimodais, Viés demográfico
Referências
Baevski, A., Zhou, H., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: a framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA. Curran Associates Inc.
Bai, Y., Hubers, F., Cucchiarini, C., and Strik, H. (2021). An asr-based reading tutor for practicing reading skills in the first grade. improving performance through threshold adjustment. ISCA.
Bailly, G., Godde, E., Piat-Marchand, A.-L., and Bosse, M.-L. (2022). Automatic assessment of oral readings of young pupils. Speech Communication, 138:67–79.
da Silva, G. C., Rodrigues, R. L., Amorim, A. N., Jeon, L., Albuquerque, E. X., Silva, V. C., da Silva, V. F., Pinheiro, A. L., Nunes, J. P., de Souza, S. X., et al. (2025). Assessing reading fluency in elementary grades: A machine learning approach. Computers and Education: Artificial Intelligence, 8:100411.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186.
Fernández-Delgado, M., Cernadas, E., Barro, S., and Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems? The journal of machine learning research, 15(1):3133–3181.
Gao, L., Tejedor-Garcia, C., Strik, H., and Cucchiarini, C. (2024). Reading miscue detection in primary school through automatic speech recognition. arXiv preprint arXiv:2406.07060.
Gou, J., Yu, B., Maybank, S. J., and Tao, D. (2021). Knowledge distillation: A survey. International journal of computer vision, 129(6):1789–1819.
Harmsen, W., van Hout, R., Cucchiarini, C., and Strik, H. (2025). Can asr generate valid measures of child reading fluency? In Proc. Interspeech 2025, pages 2395–2399.
Hasbrouck, J. and Glaser, D. R. (2012). Reading Fluency: Understanding and Teaching this Complex Skill (Training Manual). Gibson Hasbrouck & Associates.
Hayati, D. S. and Saryanto, S. (2025). Development of deep learning methods to improve reading skills for elementary school students. Jurnal Kajian Ilmu Pendidikan (JKIP), 6(3):1023–1033.
Hinton, G. E., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. ArXiv, abs/1503.02531.
LaBerge, D. and Samuels, S. J. (1974). Toward a theory of automatic information processing in reading. Cognitive psychology, 6(2):293–323.
Liu, J., Wumaier, A., Fan, C., and Guo, S. (2023a). Automatic fluency assessment method for spontaneous speech without reference text. Electronics, 12(8):1775.
Liu, W., Fu, K., Tian, X., Shi, S., Li, W., Ma, Z., and Lee, T. (2023b). An asr-free fluency scoring approach with self-supervised learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
Mohammadi, A., Koerich, A. L., Moro-Velazquez, L., and Cardinal, P. (2025). Automatic proficiency assessment in l2 english learners. arXiv preprint arXiv:2505.02615.
Petscher, Y., O'Sullivan, J., Catts, H. W., Edwards, A., and Fitton, L. (2026). Evaluation of the consistency of a speech verification system with human raters in early literacy screening assessments. In Frontiers in Education, volume 11, page 1671946. Frontiers Media SA.
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org.
Reimers, N. and Gurevych, I. (2020). Making monolingual sentence embeddings multilingual using knowledge distillation. In Webber, B., Cohn, T., He, Y., and Liu, Y., editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics.
van der Velde, M., Harmsen, W., Veldkamp, B. P., Feskens, R., Keuning, J., and Swart, N. (2025). Speech enabled reading fluency assessment: A validation study. International journal of artificial intelligence in education, 35(4):2569–2595.
van der Velde, M., Molenaar, B., Veldkamp, B. P., Feskens, R., and Keuning, J. (2024). What do they say? assessment of oral reading fluency in early primary school children: A scoping review. International journal of educational research, 128:102444.
Wang, Y., Wu, Z., Nese, J., Kamata, A., Nilabh, V., and Larson, E. C. (2025). A unified model for oral reading fluency and student prosody. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
Yıldız, M., Keskin, H. K., Oyucu, S., Hartman, D. K., Temur, M., and Aydoğmuş, M. (2025). Can artificial intelligence identify reading fluency and level? comparison of human and machine performance. Reading & Writing Quarterly, 41(1):66–83.
Zhang, J., Zhang, Z., Wang, Y., Yan, Z., Song, Q., Huang, Y., Li, K., Povey, D., and Wang, Y. (2021). speechocean762: An open-source non-native english speech corpus for pronunciation assessment. In Proc. Interspeech 2021.
Bai, Y., Hubers, F., Cucchiarini, C., and Strik, H. (2021). An asr-based reading tutor for practicing reading skills in the first grade. improving performance through threshold adjustment. ISCA.
Bailly, G., Godde, E., Piat-Marchand, A.-L., and Bosse, M.-L. (2022). Automatic assessment of oral readings of young pupils. Speech Communication, 138:67–79.
da Silva, G. C., Rodrigues, R. L., Amorim, A. N., Jeon, L., Albuquerque, E. X., Silva, V. C., da Silva, V. F., Pinheiro, A. L., Nunes, J. P., de Souza, S. X., et al. (2025). Assessing reading fluency in elementary grades: A machine learning approach. Computers and Education: Artificial Intelligence, 8:100411.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186.
Fernández-Delgado, M., Cernadas, E., Barro, S., and Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems? The journal of machine learning research, 15(1):3133–3181.
Gao, L., Tejedor-Garcia, C., Strik, H., and Cucchiarini, C. (2024). Reading miscue detection in primary school through automatic speech recognition. arXiv preprint arXiv:2406.07060.
Gou, J., Yu, B., Maybank, S. J., and Tao, D. (2021). Knowledge distillation: A survey. International journal of computer vision, 129(6):1789–1819.
Harmsen, W., van Hout, R., Cucchiarini, C., and Strik, H. (2025). Can asr generate valid measures of child reading fluency? In Proc. Interspeech 2025, pages 2395–2399.
Hasbrouck, J. and Glaser, D. R. (2012). Reading Fluency: Understanding and Teaching this Complex Skill (Training Manual). Gibson Hasbrouck & Associates.
Hayati, D. S. and Saryanto, S. (2025). Development of deep learning methods to improve reading skills for elementary school students. Jurnal Kajian Ilmu Pendidikan (JKIP), 6(3):1023–1033.
Hinton, G. E., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. ArXiv, abs/1503.02531.
LaBerge, D. and Samuels, S. J. (1974). Toward a theory of automatic information processing in reading. Cognitive psychology, 6(2):293–323.
Liu, J., Wumaier, A., Fan, C., and Guo, S. (2023a). Automatic fluency assessment method for spontaneous speech without reference text. Electronics, 12(8):1775.
Liu, W., Fu, K., Tian, X., Shi, S., Li, W., Ma, Z., and Lee, T. (2023b). An asr-free fluency scoring approach with self-supervised learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
Mohammadi, A., Koerich, A. L., Moro-Velazquez, L., and Cardinal, P. (2025). Automatic proficiency assessment in l2 english learners. arXiv preprint arXiv:2505.02615.
Petscher, Y., O'Sullivan, J., Catts, H. W., Edwards, A., and Fitton, L. (2026). Evaluation of the consistency of a speech verification system with human raters in early literacy screening assessments. In Frontiers in Education, volume 11, page 1671946. Frontiers Media SA.
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org.
Reimers, N. and Gurevych, I. (2020). Making monolingual sentence embeddings multilingual using knowledge distillation. In Webber, B., Cohn, T., He, Y., and Liu, Y., editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics.
van der Velde, M., Harmsen, W., Veldkamp, B. P., Feskens, R., Keuning, J., and Swart, N. (2025). Speech enabled reading fluency assessment: A validation study. International journal of artificial intelligence in education, 35(4):2569–2595.
van der Velde, M., Molenaar, B., Veldkamp, B. P., Feskens, R., and Keuning, J. (2024). What do they say? assessment of oral reading fluency in early primary school children: A scoping review. International journal of educational research, 128:102444.
Wang, Y., Wu, Z., Nese, J., Kamata, A., Nilabh, V., and Larson, E. C. (2025). A unified model for oral reading fluency and student prosody. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
Yıldız, M., Keskin, H. K., Oyucu, S., Hartman, D. K., Temur, M., and Aydoğmuş, M. (2025). Can artificial intelligence identify reading fluency and level? comparison of human and machine performance. Reading & Writing Quarterly, 41(1):66–83.
Zhang, J., Zhang, Z., Wang, Y., Yan, Z., Song, Q., Huang, Y., Li, K., Povey, D., and Wang, Y. (2021). speechocean762: An open-source non-native english speech corpus for pronunciation assessment. In Proc. Interspeech 2021.
Publicado
05/10/2026
Como Citar
BATISTA, Hyan H. N.; NASCIMENTO, André C. A.; MIRANDA, Péricles; MELLO, Rafael Ferreira.
Enriquecimento de Embeddings Multimodais com Representações de Encoders de Fala para Pontuação Automática de Fluência de Leitura. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 37. , 2026, Goiânia/GO.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 890-902.
DOI: https://doi.org/10.5753/sbie.2026.27401.
