ModBERTBr: A ModernBERT-based Model for Brazilian Portuguese - Wu, Wallace Ben Teng Lin; Garcia, Luis Paulo Faina (Anais do Encontro Nacional de Inteligência Artificial e Computacional (ENIAC); 2025: Anais do XXII Encontro Nacional de Inteligência Artificial e Computacional, 2025)
A key model in the Large Language Model (LLM) field is the Bidirectional Encoder Representations from Transformers (BERT), known for its effectiveness and versatility. The current state-of-the-art variant of BERT is ModernBERT, and despite excelling in efficiency and performance, it is limited to English. This paper addresses this notable gap by introducing ModBERTBr, a novel pre-trained model based on the ModernBERT architecture, which is explicitly tailored for Brazilian Portuguese and incorporates cutting-edge research and technologies. Through both intrinsic and extrinsic evaluations, ModBERTBr was assessed against multiple baseline models, showing consistent improvements and competitive performance compared to its predecessors.