Evaluating Domain-adapted Language Models for Governmental Text Classification Tasks in Portuguese - Silva, Mariana O.; Oliveira, Gabriel P.; Costa, Lucas G. L.; Pappa, Gisele L. (Anais do Simpósio Brasileiro de Banco de Dados (SBBD); 2024: Anais do XXXIX Simpósio Brasileiro de Bancos de Dados, 2024)
Domain-adaptive pre-training (DAPT) is a technique in natural language processing (NLP) that tailors pre-trained language models to specific domains, enhancing their performance in real-world applications. In this paper, we evaluate the effectiveness of DAPT in governmental text classification tasks, exploring how different factors, such as target domain dataset, pre-trained model language composition, and dataset size, impact model performance. We systematically vary these factors, creating distinct domain-adapted models derived from BERTimbau and LaBSE. Our experimental results reveal that selecting appropriate target domain datasets and pre-training strategies can notably enhance the performance of language models in governmental tasks.