Corpus-driven lexical analyses of CorCel: a comparative analysis of preliminary findings of written proficiency in Portuguese as an additional language

• Elisa Marchioro Stumpf UFRGS
• Juliana Schoffen UFRGS
• Luiza Sarmento Divino UFRGS
• Isadora Dahmer Hanauer UFRGS
• Amanda Raupp UFRGS
• Brenda Xavier UFRGS

DOI: https://doi.org/10.5753/stil.2025.37870

Abstract

This paper aims to comparatively analyze research on CorCel [Schoffen et al. , forthcoming] a corpus of written texts produced under exam conditions for the Celpe-Bras exam. It compares studies that examined tasks using Sketch Engine’s keywords, wordlist, n-grams and concordance tools. These studies also analyzed text length and lexical richness using lexical diversity indices. The comparative study showed relevant lexical indexes for characterizing the exam proficiency levels, such as text length, and important differences in the use of input material and other linguistic resources among texts rated with different grades. Such analyses offer new possibilities for research in Portuguese as an additional language (PAL) proficiency assessment and teaching.

References

Banerjee, J., Franceschina, F., and Smith, A. M. (2007). Documenting features of written language production typical at different ielts band score levels. IELTS Research Reports, 7(5):1–69.

Biber, D. and Gray, B. (2013). Discourse characteristics of writing and speaking task types on the toefl ibt® test: a lexico-grammatical analysis. ETS Research Report Series, 2013(1):i–128.

Callies, M. and Götz, S. (2015). Learner corpora in language testing and assessment: Prospects and challenges. Learner corpora in language testing and assessment, pages 1–9.

Cushing, S. T. (2017). Corpus linguistics in language testing research. Language Testing, 34(4):441–449.

Cushing, S. T. (2021). Corpus linguistics and language testing. In The Routledge Handbook of Language Testing, pages 545–560. Routledge.

Divino, L. (2021). Índices lexicais de análise para a caracterização dos níveis intermediário e avançado superior no exame celpe-bras: uma pesquisa guiada por corpus. Unpublished undergraduate thesis.

Divino, L. S. (2024). Contribuições da linguística de corpus para a descrição dos níveis de proficiência escrita no exame celpe-bras: um estudo sobre léxico. Unpublished masters thesis.

Granger, S. and Wynne, M. (2000). Optimising measures of lexical variation in efl learner corpora. In Corpora galore, pages 249–257. Brill.

Hanauer, I. (2023). Caracterização dos níveis intermediário e avançado superior do exame celpe-bras em produções escritas de examinandos no gênero carta/e-mail: contribuições de uma análise guiada por corpus. Unpublished undergraduate thesis.

INEP (2020). Documento base do exame Celpe-Bras. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira.

Kilgariff, A., Ryckly, P., Smrz, P., and Tugwell, D. (2004). The sketch engine. i: Williams, g. & s. vessier. In Proceedings of the Eleventh EURALEX International Congress, Lorient, France July 6–10, pages 105–114.

Kunrath, S. P. (2019). Os descritores gerais e a progressão dos níveis de proficiência do exame celpe-bras. Unpublished doctoral dissertation.

Mendel, K. (2019). Proficiência e autoria na avaliação integrada de leitura e escrita do exame celpe-bras. Unpublished masters thesis.

Mizumoto, A. and Eguchi, M. (2023). Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050.

Nagasawa, E. Y. (2019). O conteúdo de insumo em tarefas que integram leitura e escrita no celpe-bras: uma abordagem informada por corpus. Unpublished doctoral dissertation.

Pack, A., Barrett, A., and Escalante, J. (2024). Large language models and automated essay scoring of english language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence, 6:100234.

Paquot, M. (2019). The phraseological dimension in interlanguage complexity research. Second language research, 35(1):121–145.

Raupp, A. M. (2024). Características lexicais das produções escritas do exame celpe-bras na tarefa 3 de 2016-2: uma pesquisa guiada por corpus. Unpublished undergraduate thesis.

Rayson, P. E. (2003). Matrix: A statistical method and software tool for linguistic analysis through corpus comparison. Lancaster University (United Kingdom).

Schoffen, J., Schlatter, M., Kunrath, S. P., Nagasawa, E. Y., Sirianni, G. R., Mendel, K., Truyllio, L. R., and Divino, L. S. (2018). Estudo descritivo das tarefas da parte escrita do exame celpe-bras: Edições de 1998 a 2017. Technical report, Porto Alegre.

Schoffen, J., Stumpf, E., Amaral, D., Divino, L., Hanauer, I., Lisboa, I., Raupp, A., and Xavier, B. (2024). Compilation and tagging of a corpus with celpe-bras texts. In Proceedings of the 16th International Conference on Computational Processing of Portuguese, pages 627–632.

Schoffen, J., Stumpf, E. M., Divino, L. S., Hanauer, I. D., Amaral, D., Raupp, A., and Xavier, B. Corcel: a brazilian portuguese corpus of celpe-bras exam written texts [in press]. Revista Brasileira de Linguística Aplicada.

Sirianni, G. R. (2020). Entre a certificação e a não certificação no celpe-bras: um estudo sobre os níveis de proficiência na parte escrita do exame. Unpublished masters thesis.

Sostruznik, J. (2023). O uso de conjunções em produções escritas no exame celpe-bras: um estudo baseado em corpus. Unpublished undergraduate thesis.

Stumpf, E. M., Schoffen, J., Divino, L. S., Hanauer, I. D., Amaral, D., Raupp, A., and Xavier, B. Interrater reliability study of a tagging protocol for an l2 corpus: the case of corcel. Manuscript in preparation.

Wisniewski, K. (2017). Empirical learner language and the levels of the common european framework of reference. Language Learning, 67(S1):232–253.