United Experts: Efficient Multi-Task Sentiment Analysis in Portuguese via Model Merging

• João Assaoka UNIFESP
• Evangelos Milios UNIFESP
• Vlado Keselj UNIFESP
• José Vitor Sousa Cardoso Requena UNIFESP
• Diego de Almeida Miranda UNIFESP
• Lilian Berton UNIFESP

DOI: https://doi.org/10.5753/eniac.2026.31064

Resumo

Building robust multi-task models for sentiment analysis traditionally relies on joint training, which can be computationally prohibitive and susceptible to dataset imbalance. As an efficient alternative, we evaluate the application of model merging to combine independently trained Portuguese Small Language Models (SLMs). By fusing five distinct domain experts, we aggregate capabilities at zero additional optimization cost. Empirical results indicate that while joint training serves as a competitive empirical baseline, model merging successfully preserves individual domain expertise and offers a significantly superior compute-performance tradeoff. Additionally, we highlight the utility of merged weights as powerful prior initializations, significantly improving cross-domain transfer performance and reducing the cost of adapting the model to new, unseen domains.

Referências

Antoun, W., Baly, F., and Hajj, H. (2020). Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 9–15.

Cañete, J., Chaperon, G., Fuentes, R., Ho, J.-H., Kang, H., and Pérez, J. (2020). Spanish pre-trained bert model and evaluation data. In PML4DC at ICLR 2020.

Corrêa, N. K., Sen, A., Fatimah, S., Falk, S., Landgraf, L., Kastner, J., and Flek, L. (2026). Tucano 2 cool: Better open source llms for portuguese.

Crawshaw, M. (2020). Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796.

Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., and Hu, G. (2020). Revisiting pre-trained models for chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 657–668.

Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., and Ravi, S. (2020). GoEmotions: A dataset of fine-grained emotions. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, Online. Association for Computational Linguistics.

Freitas, C., Motta, E., Milidiú, R. L., and César, J. (2014). Sparkling vampire... lol! annotating opinions in a book review corpus. In Aluísio, S. and Tagnin, S. E. O., editors, New Language Technologies and Linguistic Research: A Two-Way Road, pages 128–146. Cambridge Scholars Publishing.

French, R. M. (1999). Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135.

Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L., Karpukhin, V., Benedict, B., McQuade, M., and Solawetz, J. (2025). Arcee’s mergekit: A toolkit for merging large language models.

Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR.

Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. (2023). Editing models with task arithmetic.

Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526.

Liu, B. (2020). Sentiment analysis: mining opinions, sentiments, and emotions. Cambridge university press.

Liu, X., He, P., Chen, W., and Gao, J. (2019). Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487–4496.

Maddigan, P. et al. (2024). The rise of small language models: A survey of efficient architectures and applications. arXiv preprint.

Malo, P., Sinha, A., Korhonen, P. J., Wallenius, J., and Takala, P. (2013). Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65.

Muhammad, S. H. et al. (2025). Brighter: Bridging the gap in human-annotated textual emotion recognition datasets for 28 languages. arXiv preprint arXiv:2502.11926.

Santos, L. L., Bianchi, R. A., and Costa, A. H. (2023). Finbert-pt-br: Análise de sentimentos de textos em português do mercado financeiro. In Anais do II Brazilian Workshop on Artificial Intelligence in Finance, pages 144–155. SBC.

Souza, F., Nogueira, R., and Lotufo, R. (2020). Bertimbau: pretrained language models for brazilian portuguese. In Brazilian Conference on Intelligent Systems, pages 403–411. Springer.

Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. (2022). Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.

Yadav, P., Tam, D., Choshen, L., Raffel, C., and Bansal, M. (2023). Ties-merging: Resolving interference when merging models.

Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y. (2024). Language models are super mario: Absorbing abilities from homologous models as a free lunch.

Zhang, L., Wang, S., and Liu, B. (2018). Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1253.