Predição de transtorno depressivo em redes sociais: BERT supervisionado ou ChatGPT zero-shot?

Resumo


Este artigo apresenta um primeiro estudo sobre o uso do sistema de dialogo ChatGPT em uma aplicação complexa e sensível: a predição computacional de transtornos de saúde mental a partir de textos provenientes de redes sociais. Para esse fim, foi conduzido um experimento comparando uma abordagem supervisionada tradicional baseada em BERT com uma estratégia zero-shot baseada em prompts em língua natural submetidos diretamente ao sistema de diálogo. Resultados desta avaliação, levando em conta a acurácia da tarefa de classificação face à necessidade de anotação prévia de córpus da abordagem supervisionada, destacam diferentes vantagens de cada alternativa.

Palavras-chave: saúde mental, depressão, redes sociais, BERT, ChatGPT

Referências

Ansari, L. and Ji, S. (2022). Ensemble hybrid learning methods for automated depression detection. IEEE Transactions on computational Social Systems https://dx.doi.org/10.1109/TCSS.2022.3154442

Aragón, M. E., López-Monroy, A. P., González-Gurrola, L. C., and y Gómez, M. M. (2019). Detecting depression in social media using fine-grained emotions. In NAACL-2019 Proceedings, pages 1481–1486, Minneapolis, USA. Assoc for Comp Ling. http://dx.doi.org/10.18653/v1/N19-1151

BigScience Workshop (2022). BLOOM (revision 4ab0472). https://arxiv.org/pdf/2211.05100.pdf

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. In NIPS’20: Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 1877–1901 https://doi.org/10.48550/arXiv.2005.14165

Burdisso, S. G., Errecalde, M., and y Gómez, M. M. (2020). t-SS3: a text classifier with dynamic n-grams for early risk detection over text streams. Pattern Recognition Letters, 138:130–137. https://doi.org/10.1016/j.patrec.2020.07.001

Cacheda, F., Fernandez, D., Novoa, F. J., and Carneiro, V. (2019). Early detection of depression: Social network analysis and random forest techniques. J Med Internet Res, 21(6):e12554. https://doi.org/10.2196/12554

Chancellor, S. and Choudhury, M. D. (2020). Methods in predictive techniques for mental health status on social media: a critical review. npj Digit. Med., 3(43). https://doi.org/10.1038/s41746-020-0233-7

Cohan, A., Desmet, B., Yates, A., Soldaini, L., MacAvaney, S., and v Goharian (2018). SMHD: a large-scale resource for exploring online language usage for multiple mental health conditions. In COLING-2018, pages 1485–1497, Santa Fe, USA https://aclanthology.org/C18-1126

Coppersmith, G., Dredze, M., Harman, C., Kristy, H., and Mitchell, M. (2015). CLPsych 2015 Shared Task: Depression and PTSD on Twitter. In 2nd Workshop on Computational Linguistics and Clinical Psychology, pages 31–39, Denver, USA http://dx.doi.org/10.3115/v1/W15-1204

da Costa, P. B., Pavan, M. C., dos Santos, W. R., da Silva, S. C., and Paraboni, I. (2023). BERTabaporu: assessing a genre-specific language model for Portuguese NLP. In Recents Advances in Natural Language Processing (RANLP-2023), Varna, Bulgaria [link].

da Silva, S. C., Ferreira, T. C., Ramos, R. M. S., and Paraboni, I. (2020). Data driven and psycholinguistics motivated approaches to hate speech detection. Computacion y Systemas, 24(3):1179–1188 https://doi.org/10.13053/cys-24-3-3478

Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT 2019 Proceedings, pages 4171–4186, Minneapolis, USA http://dx.doi.org/10.18653/v1/N19-1423

dos Santos, W. R., de Oliveira, R. L., and Paraboni, I. (2023). SetembroBR: a social media corpus for depression and anxiety disorder prediction. Language Resources and Evaluation https://doi.org/10.1007/s10579-022-09633-0

dos Santos, W. R., Funabashi, A. M. M., and Paraboni, I. (2020a). Searching Brazilian Twitter for signs of mental health issues. In 12th International Conference on Language Resources and Evaluation (LREC-2020), pages 6113–6119, Marseille, France https://aclanthology.org/2020.lrec-1.750

dos Santos, W. R., Ramos, R. M. S., and Paraboni, I. (2020b). Computational personality recognition from facebook text: psycholinguistic features, words and facets. New Review of Hypermedia and Multimedia, 25(4):268–287 https://doi.org/10.1080/13614568.2020.1722761

Flores, A. M., Pavan, M. C., and Paraboni, I. (2022). User profiling and satisfaction inference in public information access services. Journal of Intelligent Information Systems, 58(1):67–89 https://doi.org/10.1007/s10844-021-00661-w

Kumar, A., Sharma, A., and Arora, A. (2019). Anxious depression prediction in real-time social data. In Intl. Conf. on Advances in Engineering Science Management & Technology, Dehradun, India [link].

Kuzman, T., Mozetič, I., and Ljubešić, N. (2023). ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification. arXiv preprint arXiv:2303.03953 https://doi.org/10.48550/arXiv.2303.03953

Lin, C., Hu, P., Su, H., Li, S., Mei, J., Zhou, J., and Leung, H. (2020). SenseMood: Depression Detection on Social Media, pages 407–411. Association for Computing Machinery, New York, USA https://doi.org/10.1145/3372278.3391932

Losada, D. E. and Crestani, F. (2016). A test collection for research on depression and language use. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 28–39, Cham. Springer. https://doi.org/10.1007/978-3-319-44564-9_3

Losada, D. E., Crestani, F., and Parapar, J. (2017). eRISK 2017: CLEF lab on early risk prediction on the internet: experimental foundations. In LNCS 10456, pages 346–360, Cham. Springer. https://doi.org/10.1007/978-3-319-65813-1_30

Losada, D. E., Crestani, F., and Parapar, J. (2019). Overview of eRisk 2019 Early Risk Prediction on the Internet. In LNCS 11696. https://doi.org/10.1007/978-3-030-28577-7_27

Lynn, V., Goodman, A., Niederhoffer, K., Loveys, K., Resnik, P., and Schwartz, H. A. (2018). CLPsych 2018 shared task: Predicting current and future psychological health from childhood essays. In Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, pages 37–46, New Orleans, USA http://dx.doi.org/10.18653/v1/W18-0604

Mikolov, T., Wen-tau, S., and Zweig, G. (2013). Linguistic regularities in continuous space word representations. In Proc. of NAACL-HLT-2013, pages 746–751, Atlanta, USA. Assoc for Comp Ling https://aclanthology.org/N13-1090.pdf

Parapar, J., Martin-Rodilla, P., Losada, D. E., and Crestani, F. (2022). Overview of eRisk 2022: Early Risk Prediction on the Internet. In Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, pages 821–850, Bologna, Italy https://ceur-ws.org/Vol-3180/paper-66.pdf

Pavan, M. C., dos Santos, V. G., Lan, A. G. J., ao Trevisan Martins, J., dos Santos, W. R., Deutsch, C., da Costa, P. B., Hsieh, F. C., and Paraboni, I. (2023). Morality classification in natural language text. IEEE transactions on Affective Computing, 14(1):857–863 https://doi.org/10.1109/TAFFC.2020.3034050

Pavan, M. C., dos Santos, W. R., and Paraboni, I. (2020). Twitter Moral Stance Classification using Long Short-Term Memory Networks. In 9th Brazilian Conference on Intelligent Systems (BRACIS). LNAI 12319, pages 636–647. Springer https://doi.org/10.1007/978-3-030-61377-8_45

Pavan, M. C. and Paraboni, I. (2022). Cross-target stance classification as domain adaptation. In Pichardo Lagunas, O., Martınez-Miranda, J., and Martınez Seis, B., editors, Advances in Computational Intelligence - MICAI 2022 - Lecture Notes in Artificial Intelligence vol 13612, pages 15–25, Cham. Springer Nature Switzerland https://doi.org/10.1007/978-3-031-19493-1_2

Pennebaker, J. W., Francis, M. E., and Booth, R. J. (2001). Inquiry and Word Count: LIWC. Lawrence Erlbaum, Mahwah, NJ.

Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., and Yang, D. (2023). Is ChatGPT a General-Purpose Natural Language Processing Task Solver? arXiv preprint arXiv:2302.06476 https://doi.org/10.48550/arXiv.2302.06476

Souza, V., Nobre, J., and Becker, K. (2020). Characterization of anxiety, depression, and their comorbidity from texts of social networks. In SBBD-2020, pages 121–132, Porto Alegre, Brazil. SBC https://doi.org/10.5753/sbbd.2020.13630

Souza, V., Nobre, J., and Becker, K. (2021). A deep learning ensemble to classify anxiety, depression, and their comorbidity from texts of social networks. Journal of Information and Data Management, 12(3):306–325 https://doi.org/10.5753/jidm.2021.1901

Su, C., Xu, Z., Pathak, J., and Wang, F. (2020). Deep learning in mental health outcome research: a scoping review. Translational Psychiatry, 10(116) https://doi.org/10.1038/s41398-020-0780-3

Trotzek, M., Koitka, S., and Friedrich, C. M. (2018). Utilizing neural networks and linguistic metadata for early detection of depression indications in text sequences. IEEE Transactions on Knowledge and Data Engineering https://doi.org/10.1109/TKDE.2018.2885515

Yazdavar, A. H., Mahdavinejad, M. S., Bajaj, G., Romine, W., Sheth, A., Monadjemi, A. H., Thirunarayan, K., Meddar, J. M., Myers, A., Pathak, J., and Hitzler, P. (2020). Multimodal mental health analysis in social media. PLOS ONE, 15(4):1–27 https://doi.org/10.1371/journal.pone.0226248

Zhang, B., Ding, D., and Jing, L. (2023). How would Stance Detection Techniques Evolve after the Launch of ChatGPT? arXiv preprint arXiv:2212.14548 https://doi.org/10.48550/arXiv.2212.14548
Publicado
25/09/2023
DOS SANTOS, Wesley Ramos; PARABONI, Ivandré. Predição de transtorno depressivo em redes sociais: BERT supervisionado ou ChatGPT zero-shot?. In: SIMPÓSIO BRASILEIRO DE TECNOLOGIA DA INFORMAÇÃO E DA LINGUAGEM HUMANA (STIL), 14. , 2023, Belo Horizonte/MG. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2023 . p. 11-21. DOI: https://doi.org/10.5753/stil.2023.233275.