Evaluating Language Models' capacity for sentiment analysis in mental health contexts
Abstract
Mental disorders negatively impact peoples’ quality of life. These conditions manifest through sentiments such as sadness, loneliness, apathy, and fear. Thus, sentiment analysis can enhance the early identification of mental symptoms. Therefore, this article evaluates the capacity of language models based on Artificial Intelligence concerning the sentiment analysis of texts in Brazilian Portuguese. The evaluation process occurred based on a dataset of 5,000 text posts from Twitter. The caramelo-smile-2 model obtained the best performance in the experiment, reaching approximately 0.8 of precision, recall, and f1-score. The results highlight the potential of using pre-trained language models for sentiment analysis of Brazilian Portuguese texts, which can allow the utilization of these models in applications to analyze text messages of patients diagnosed with mental disorders.
References
Adilmar Coelho Dantas (2025). caramelo-smile (revision 2707a19). DOI: 10.57967/hf/2061. Disponível em: [link]. Acessado em: 24-02-2025.
Almeida, C., Castro, C., Leiva, V., Braga, A. C., and Freitas, A. (2024). Optimizing sentiment analysis models for customer support: Methodology and case study in the portuguese retail sector. Journal of Theoretical and Applied Electronic Commerce Research, 19(2):1493–1516. DOI: 10.3390/jtaer19020074.
American Psychiatric Association (2022). Diagnostic and Statistical Manual of Mental Disorders - Text Revision (DSM-5-TR). American Psychiatric Association Publishing, 5th edition. DOI: 10.1176/appi.books.9780890425787.
Beck, A. T. and Alford, B. A. (2008). Depression. University of Pennsylvania Press, Baltimore, MD, 2nd edition.
Bengio, Y., Lecun, Y., and Hinton, G. (2021). Deep learning for ai. Communications of the ACM, 64(7):58–65. DOI: 10.1145/3448250.
Birjali, M., Kasri, M., and Beni-Hssane, A. (2021). A comprehensive survey on sentiment analysis: Approaches, challenges and trends. Knowledge-Based Systems, 226:107134. DOI: 10.1016/j.knosys.2021.107134.
Bluhm, T. D. F. (2025). clf-sentimentos-cmts. Disponível em: [link]. Acessado em: 24-02-2025.
Branco, A., Parada, D., Silva, M., Mendonca, F., Mostafa, S. S., and Morgado-Dias, F. (2024). Sentiment analysis in portuguese restaurant reviews: Application of transformer models in edge computing. Electronics, 13(3). DOI: 10.3390/electronics13030589.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. DOI: 10.48550/ARXIV.2005.14165.
Cardoso, M. H., Maria da Rocha Fernandes, A., Marin, G., Quietinho Leithardt, V. R., and Crocker, P. (2021). Comparison between different approaches to sentiment analysis in the context of the portuguese language. In 16th Iberian Conference on Information Systems and Technologies (CISTI), pages 1–6. DOI: 10.23919/CISTI52073.2021.9476501.
Carosia, A. E. O., Coelho, G. P., and Silva, A. E. A. (2020). Analyzing the brazilian financial market through portuguese sentiment analysis in social media. Applied Artificial Intelligence, 34(1):1–19. DOI: 10.1080/08839514.2019.1673037.
Carvalho, I., Oliveira, H. G., and Silva, C. (2023). The importance of context for sentiment analysis in dialogues. IEEE Access, 11:86088–86103. DOI: 10.1109/ACCESS.2023.3304633.
Chaves, I. C., Martins, A. D. F., Praciano, F. D. B. S., Brito, F. T., Monteiro, J. M., and Machado, J. C. (2022). Bpa: A multilingual sentiment analysis approach based on bilstm. In Filipe, J., Smialek, M., Brodsky, A., and Hammoudi, S., editors, Proceedings of the 24th International Conference on Enterprise Information Systems (ICEIS), volume 1, pages 553–560. INSTICC, SCITEPRESS - Science and Technology Publications. DOI: 10.5220/0011071400003179.
Corrêa, E. A., Marinho, V. Q., dos Santos, L. B., Fe C. Bertaglia, T., Treviso, M. V., and Brum, H. B. (2017). Pelesent: Cross-domain polarity classification using distant supervision. In 2017 Brazilian Conference on Intelligent Systems (BRACIS), pages 49–54. DOI: 10.1109/BRACIS.2017.45.
da Silva, A. N., de Souza, O., and de Souza, J. N. (2020). Sentiment parser based on x-bar theory to brazilian portuguese. In 2020 International Conference on Computing, Electronics Communications Engineering (iCCECE), pages 166–171. DOI: 10.1109/iCCECE49321.2020.9231145.
da Silva, F. C. D., Garcia, A. C. B., and Siqueira, S. W. M. (2023). Sentiment gradient-improving sentiment analysis with entropy increase. Inteligencia Artificial - Iberoamerical Journal of Artificial Intelligence, 26(71):114–130. DOI: 10.4114/intartif.vol26iss71pp114-130.
Go, A., Bhayani, R., and Huang, L. (2009). Twitter sentiment classification using distant supervision. In Proceedings of the International Conference on Machine Learning. Disponível em: [link]. Acessado em: 24-02-2025.
Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on tabular data? DOI: 10.48550/ARXIV.2207.08815.
Heckler, W. F., de Carvalho, J. V., and Barbosa, J. L. V. (2022). Machine learning for suicidal ideation identification: A systematic literature review. Computers in Human Behavior, 128.
Heckler, W. F., Feijó, L. P., de Carvalho, J. V., and Barbosa, J. L. V. (2023). Thoth: An intelligent model for assisting individuals with suicidal ideation. Expert Systems with Applications, 233:120918.
Heckler, W. F., Feijó, L. P., de Carvalho, J. V., and Barbosa, J. L. V. (2025). Digital phenotyping for mental health based on data analytics: A systematic literature review. Artificial Intelligence in Medicine, 163:103094.
Hugging Face (2025a). Distilbert model documentation. Disponível em: [link]. Acessado em: 24-02-2025.
Hugging Face (2025b). Xlm-roberta model documentation. Disponível em: [link]. Acessado em: 24-02-2025.
Kaggle (2025). Portuguese tweets for sentiment analysis. Disponível em: [link]. Acessado em: 24-02-2025.
Kouloumpis, E., Wilson, T., and Moore, J. (2021). Twitter Sentiment Analysis: The Good the Bad and the OMG! Proceedings of the International AAAI Conference on Web and Social Media, 5(1):538–541. DOI: 10.1609/icwsm.v5i1.14185.
Lopes, F. C., da Costa, M. P., Fernandez-Lazaro, I, C., Lara-Abelenda, F. J., Pereira-Sanchez, V., Teo, A. R., and Alvarez-Mon, M. A. (2024). Analysis of the hikikomori phenomenon - an international infodemiology study of twitter data in portuguese. BMC Public Health, 24(1). DOI: 10.1186/s12889-023-17617-0.
Marques, T., Cezario, S., Lacerda, J., Pinto, R., Silva, L., Santana, O., Ribeiro, A. G., Cruz, A. S., Miranda, A. E., Cadaxa, A., Nunez, L. S., Oliveira, H. G., Atun, R., and Valentim, R. (2022). Sentiment analysis in understanding the potential of online news in the public health crisis response. International Journal of Environmental Research and Public Health, 19(24). DOI: 10.3390/ijerph192416801.
Martins, R., Almeida, J. J., Henriques, P., and Novais, P. (2021). A sentiment analysis approach to increase authorship identification. Expert Systems, 38(5). DOI: 10.1111/exsy.12469.
Moreira, V. D. S., Siqueira, S. W., Andrade, L., and Pimentel, M. (2016). Análise de sentimentos: Comparando o uso de ferramentas e a análise humana. In Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI), SBSI 2016, page 441–448. Sociedade Brasileira de Computação.
OpenAI (2024). Gpt-4o mini: Modelo de linguagem multimodal compacto. Disponível em: [link]. Acessado em: 24-02-2025.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. Technical report, OpenAI.
Raychawdhary, N., Das, A., Bhattacharya, S., Dozier, G., and Seals, C. D. (2024). Optimizing multilingual sentiment analysis in low-resource languages with adaptive pre-training and strategic language selection. In Abdelgawad, A., Jamil, A., and Hameed, A., editors, IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI). IEEE. DOI: 10.1109/ICMI60790.2024.10585876.
Sakiyama, K. M., Silva, A. Q. B., and Matsubara, E. T. (2019). Twitter breaking news detector in the 2018 brazilian presidential election using word embeddings and convolutional neural networks. In International Joint Conference on Neural Networks (IJCNN), pages 1–8. DOI: 10.1109/IJCNN.2019.8852394.
Souza, F. D. and Baptista de Oliveira e Souza Filho, J. (2021). Sentiment analysis on brazilian portuguese user reviews. In IEEE Latin American Conference on Computational Intelligence (LA-CCI), pages 1–6. DOI: 10.1109/LA-CCI48322.2021.9769838.
Statista (2024). Prevalence of mental health conditions in Brazil as of 2022, by selected category. Disponível em: [link]. Acessado em: 24-02-2025.
TabularisAI (2025). multilingual-sentiment-analysis. Disponível em: [link]. Acessado em: 24-02-2025.
Wang, R. (2025). distilbert-base-uncased-finetuned-sst2-pt. Disponível em: [link]. Acessado em: 24-02-2025.
World Health Organization (2021). Mental Health Action Plan 2013-2030 flyer: what Member States can do. Disponível em: [link]. Acessado em: 24-02-2025.
Yuan, L. X. (2025). distilbert-base-multilingual-cased-sentiments-student. Disponível em: [link]. Acessado em: 24-02-2025.
Almeida, C., Castro, C., Leiva, V., Braga, A. C., and Freitas, A. (2024). Optimizing sentiment analysis models for customer support: Methodology and case study in the portuguese retail sector. Journal of Theoretical and Applied Electronic Commerce Research, 19(2):1493–1516. DOI: 10.3390/jtaer19020074.
American Psychiatric Association (2022). Diagnostic and Statistical Manual of Mental Disorders - Text Revision (DSM-5-TR). American Psychiatric Association Publishing, 5th edition. DOI: 10.1176/appi.books.9780890425787.
Beck, A. T. and Alford, B. A. (2008). Depression. University of Pennsylvania Press, Baltimore, MD, 2nd edition.
Bengio, Y., Lecun, Y., and Hinton, G. (2021). Deep learning for ai. Communications of the ACM, 64(7):58–65. DOI: 10.1145/3448250.
Birjali, M., Kasri, M., and Beni-Hssane, A. (2021). A comprehensive survey on sentiment analysis: Approaches, challenges and trends. Knowledge-Based Systems, 226:107134. DOI: 10.1016/j.knosys.2021.107134.
Bluhm, T. D. F. (2025). clf-sentimentos-cmts. Disponível em: [link]. Acessado em: 24-02-2025.
Branco, A., Parada, D., Silva, M., Mendonca, F., Mostafa, S. S., and Morgado-Dias, F. (2024). Sentiment analysis in portuguese restaurant reviews: Application of transformer models in edge computing. Electronics, 13(3). DOI: 10.3390/electronics13030589.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. DOI: 10.48550/ARXIV.2005.14165.
Cardoso, M. H., Maria da Rocha Fernandes, A., Marin, G., Quietinho Leithardt, V. R., and Crocker, P. (2021). Comparison between different approaches to sentiment analysis in the context of the portuguese language. In 16th Iberian Conference on Information Systems and Technologies (CISTI), pages 1–6. DOI: 10.23919/CISTI52073.2021.9476501.
Carosia, A. E. O., Coelho, G. P., and Silva, A. E. A. (2020). Analyzing the brazilian financial market through portuguese sentiment analysis in social media. Applied Artificial Intelligence, 34(1):1–19. DOI: 10.1080/08839514.2019.1673037.
Carvalho, I., Oliveira, H. G., and Silva, C. (2023). The importance of context for sentiment analysis in dialogues. IEEE Access, 11:86088–86103. DOI: 10.1109/ACCESS.2023.3304633.
Chaves, I. C., Martins, A. D. F., Praciano, F. D. B. S., Brito, F. T., Monteiro, J. M., and Machado, J. C. (2022). Bpa: A multilingual sentiment analysis approach based on bilstm. In Filipe, J., Smialek, M., Brodsky, A., and Hammoudi, S., editors, Proceedings of the 24th International Conference on Enterprise Information Systems (ICEIS), volume 1, pages 553–560. INSTICC, SCITEPRESS - Science and Technology Publications. DOI: 10.5220/0011071400003179.
Corrêa, E. A., Marinho, V. Q., dos Santos, L. B., Fe C. Bertaglia, T., Treviso, M. V., and Brum, H. B. (2017). Pelesent: Cross-domain polarity classification using distant supervision. In 2017 Brazilian Conference on Intelligent Systems (BRACIS), pages 49–54. DOI: 10.1109/BRACIS.2017.45.
da Silva, A. N., de Souza, O., and de Souza, J. N. (2020). Sentiment parser based on x-bar theory to brazilian portuguese. In 2020 International Conference on Computing, Electronics Communications Engineering (iCCECE), pages 166–171. DOI: 10.1109/iCCECE49321.2020.9231145.
da Silva, F. C. D., Garcia, A. C. B., and Siqueira, S. W. M. (2023). Sentiment gradient-improving sentiment analysis with entropy increase. Inteligencia Artificial - Iberoamerical Journal of Artificial Intelligence, 26(71):114–130. DOI: 10.4114/intartif.vol26iss71pp114-130.
Go, A., Bhayani, R., and Huang, L. (2009). Twitter sentiment classification using distant supervision. In Proceedings of the International Conference on Machine Learning. Disponível em: [link]. Acessado em: 24-02-2025.
Grinsztajn, L., Oyallon, E., and Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on tabular data? DOI: 10.48550/ARXIV.2207.08815.
Heckler, W. F., de Carvalho, J. V., and Barbosa, J. L. V. (2022). Machine learning for suicidal ideation identification: A systematic literature review. Computers in Human Behavior, 128.
Heckler, W. F., Feijó, L. P., de Carvalho, J. V., and Barbosa, J. L. V. (2023). Thoth: An intelligent model for assisting individuals with suicidal ideation. Expert Systems with Applications, 233:120918.
Heckler, W. F., Feijó, L. P., de Carvalho, J. V., and Barbosa, J. L. V. (2025). Digital phenotyping for mental health based on data analytics: A systematic literature review. Artificial Intelligence in Medicine, 163:103094.
Hugging Face (2025a). Distilbert model documentation. Disponível em: [link]. Acessado em: 24-02-2025.
Hugging Face (2025b). Xlm-roberta model documentation. Disponível em: [link]. Acessado em: 24-02-2025.
Kaggle (2025). Portuguese tweets for sentiment analysis. Disponível em: [link]. Acessado em: 24-02-2025.
Kouloumpis, E., Wilson, T., and Moore, J. (2021). Twitter Sentiment Analysis: The Good the Bad and the OMG! Proceedings of the International AAAI Conference on Web and Social Media, 5(1):538–541. DOI: 10.1609/icwsm.v5i1.14185.
Lopes, F. C., da Costa, M. P., Fernandez-Lazaro, I, C., Lara-Abelenda, F. J., Pereira-Sanchez, V., Teo, A. R., and Alvarez-Mon, M. A. (2024). Analysis of the hikikomori phenomenon - an international infodemiology study of twitter data in portuguese. BMC Public Health, 24(1). DOI: 10.1186/s12889-023-17617-0.
Marques, T., Cezario, S., Lacerda, J., Pinto, R., Silva, L., Santana, O., Ribeiro, A. G., Cruz, A. S., Miranda, A. E., Cadaxa, A., Nunez, L. S., Oliveira, H. G., Atun, R., and Valentim, R. (2022). Sentiment analysis in understanding the potential of online news in the public health crisis response. International Journal of Environmental Research and Public Health, 19(24). DOI: 10.3390/ijerph192416801.
Martins, R., Almeida, J. J., Henriques, P., and Novais, P. (2021). A sentiment analysis approach to increase authorship identification. Expert Systems, 38(5). DOI: 10.1111/exsy.12469.
Moreira, V. D. S., Siqueira, S. W., Andrade, L., and Pimentel, M. (2016). Análise de sentimentos: Comparando o uso de ferramentas e a análise humana. In Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI), SBSI 2016, page 441–448. Sociedade Brasileira de Computação.
OpenAI (2024). Gpt-4o mini: Modelo de linguagem multimodal compacto. Disponível em: [link]. Acessado em: 24-02-2025.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. Technical report, OpenAI.
Raychawdhary, N., Das, A., Bhattacharya, S., Dozier, G., and Seals, C. D. (2024). Optimizing multilingual sentiment analysis in low-resource languages with adaptive pre-training and strategic language selection. In Abdelgawad, A., Jamil, A., and Hameed, A., editors, IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI). IEEE. DOI: 10.1109/ICMI60790.2024.10585876.
Sakiyama, K. M., Silva, A. Q. B., and Matsubara, E. T. (2019). Twitter breaking news detector in the 2018 brazilian presidential election using word embeddings and convolutional neural networks. In International Joint Conference on Neural Networks (IJCNN), pages 1–8. DOI: 10.1109/IJCNN.2019.8852394.
Souza, F. D. and Baptista de Oliveira e Souza Filho, J. (2021). Sentiment analysis on brazilian portuguese user reviews. In IEEE Latin American Conference on Computational Intelligence (LA-CCI), pages 1–6. DOI: 10.1109/LA-CCI48322.2021.9769838.
Statista (2024). Prevalence of mental health conditions in Brazil as of 2022, by selected category. Disponível em: [link]. Acessado em: 24-02-2025.
TabularisAI (2025). multilingual-sentiment-analysis. Disponível em: [link]. Acessado em: 24-02-2025.
Wang, R. (2025). distilbert-base-uncased-finetuned-sst2-pt. Disponível em: [link]. Acessado em: 24-02-2025.
World Health Organization (2021). Mental Health Action Plan 2013-2030 flyer: what Member States can do. Disponível em: [link]. Acessado em: 24-02-2025.
Yuan, L. X. (2025). distilbert-base-multilingual-cased-sentiments-student. Disponível em: [link]. Acessado em: 24-02-2025.
Published
2025-06-09
How to Cite
HENZ, Miguel D.; HECKLER, Wesllei F.; BARBOSA, Jorge L. V..
Evaluating Language Models' capacity for sentiment analysis in mental health contexts. In: BRAZILIAN SYMPOSIUM ON COMPUTING APPLIED TO HEALTH (SBCAS), 25. , 2025, Porto Alegre/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 293-304.
ISSN 2763-8952.
DOI: https://doi.org/10.5753/sbcas.2025.7075.
