Optimizing Model Merging Configurations for Brazilian Portuguese Sentiment and Hate Speech Classification with TIES-Merging and SaDE

• Viviane Galvão UFJF
• Heder Bernardino UFJF

Resumo

The detection of emotions and opinions from textual data plays a critical role in diverse social applications, including political analysis, content moderation, online safety assurance, and the monitoring of emotional well-being in healthcare contexts. Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing (NLP) tasks, such as sentiment and hate speech detection. However, their effectiveness in specialized domains remains a challenge, often requiring adaptation techniques such as fine-tuning to optimize performance. For Brazilian Portuguese, sentiment and hate speech classification are tasks less explored compared to English, emphasizing the need for efficient adaptation strategies. Model merging methods have emerged as promising alternatives to obtain new language models without incurring the high computational costs or dataset requirements associated with fine-tuning techniques. This study investigates the integration of Self-adaptive Differential Evolution (SaDE) with the TIES-Merging method to optimize merging configurations of BERTimbau and its fine-tuned versions into a single model for Brazilian Portuguese sentiment and hate speech classification. Experimental results show that applying TIES-Merging, supported by evolutionary methods, produces models that outperform the existing fine-tuned model in the Brazilian Portuguese sentiment classification, while maintaining competitive performance in hate speech detection. In the comparison of evolutionary strategies, SaDE achieved results comparable to CMA-ES, a method commonly used in the literature, highlighting opportunities for further investigation into the tuning of the learning period parameter.