Abordagem Semi-Supervisionada para Anotação de Linguagem Tóxica
Resumo
Mensagens tóxicas acarretam sérios problemas nas plataformas de redes sociais, uma vez que são usadas para prejudicar indivíduos, grupos ou organizações. Os métodos automáticos de combate ao Discurso de Ódio precisam de bons recursos linguísticos, como corpora. A construção manual de corpus de linguagem tóxica impõe desafios significativos devido à forte subjetividade associada ao conceito de Discurso de Ódio e à dificuldade em treinar adequadamente anotadores. A solução deste problema passa pela criação de alternativas para a anotação de dados. Este trabalho apresenta uma técnica semi-supervisionada, baseada em grafo heterogêneo, para detecção e anotação automática de linguagem tóxica. Essa abordagem foi avaliada sobre o corpus ToLD-BR e apresentou nível de concordância moderada com seus rótulos originais.Referências
Aroyo, L., Dixon, L., Thain, N., Redfield, O., and Rosen, R. (2019). Crowdsourcing subjective tasks: The case study of understanding toxicity in online discussions. In Companion Proceedings of The 2019 World Wide Web Conference, WWW ’19, page 1100–1105, New York, NY, USA. Association for Computing Machinery.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1877–1901, Online. Curran Associates, Inc.
Costa Bertaglia, T. F. and Volpe Nunes, M. d. G. (2016). Exploring word embeddings for unsupervised textual user-generated content normalization. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT), pages 112–120, Osaka, Japan. The COLING 2016 Organizing Committee.
de Pelle, R. and Moreira, V. (2017). Offensive comments in the brazilian web: a dataset and baseline results. In VI Brazilian Workshop on Social Network Analysis and Mining, pages 510–519, São Paulo, Brazil. SBC.
Fortuna, P., Rocha da Silva, J., Soler-Company, J., Wanner, L., and Nunes, S. (2019). A hierarchically-labeled Portuguese hate speech dataset. In Roberts, S. T., Tetreault, J., Prabhakaran, V., and Waseem, Z., editors, Proceedings of the Third Workshop on Abusive Language Online, pages 94–104, Florence, Italy. Association for Computational Linguistics.
Hartmann, N., Fonseca, E., Shulby, C., Treviso, M., Silva, J., and Aluísio, S. (2017). Portuguese word embeddings: Evaluating on word analogies and natural language tasks. In Proceedings of the 11th Brazilian Symposium in Information and Human Language Technology, pages 122–131, Uberlândia, Brazil. Sociedade Brasileira de Computação.
Hettiachchi, D., Holcombe-James, I., Livingstone, S., de Silva, A., Lease, M., Salim, F., and Sanderson, M. (2023). How crowd worker factors influence subjective annotations: A study of tagging misogynistic hate speech in tweets. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 11:38–50.
Lees, A., Tran, V. Q., Tay, Y., Sorensen, J., Gupta, J., Metzler, D., and Vasserman, L. (2022). A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, page 3197–3207, New York, NY, USA. Association for Computing Machinery.
Leite, J. A., Silva, D., Bontcheva, K., and Scarton, C. (2020). Toxic language detection in social media for Brazilian Portuguese: New dataset and multilingual analysis. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 914–924, Suzhou, China. Association for Computational Linguistics.
Nascimento, F. R., Cavalcanti, G. D., and Da Costa-Abreu, M. (2022). Unintended bias evaluation: An analysis of hate speech detection and gender bias mitigation on social media using ensemble learning. Expert Systems with Applications, 201:117032.
Oliveira, A., Cecote, T., Silva, P., Castro Gertrudes, J., Freitas, V., and Luz, E. (2023). How good is chatgpt for detecting hate speech in portuguese? pages 94–103.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830.
Ross, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., and Wojatzki, M. (2016). Measuring the reliability of hate speech annotations: The case of the european refugee crisis. In Dipper, S., editor, NLP4CMC III: 3rd Workshop on Natural Language Processing for Computer-Mediated Communication, Bochumer Linguistische Arbeitsberichte, pages 6–9, Germany. Ruhr-Universitat Bochum.
Rossi, R. G. (2015). Classificação automática de textos por meio de aprendizado de máquina baseado em redes. PhD thesis, Instituto de Ciências Matemáticas e de Computação.
Russell, S. J. and Norvig, P. (2009). Artificial Intelligence: a modern approach. Pearson, 3 edition.
Saraiva, G. D., Anchiêta, R., Neto, F. A. R., and Moura, R. (2021). A semi-supervised approach to detect toxic comments. In Mitkov, R. and Angelova, G., editors, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1261–1267, Held Online. INCOMA Ltd.
Souza, F., Nogueira, R., and Lotufo, R. (2020). Bertimbau: Pretrained bert models for brazilian portuguese. In Cerri, R. and Prati, R. C., editors, Proceedings of the 9th Brazilian Conference on Intelligent Systems, pages 403–417, Rio Grande. Springer International Publishing.
Vargas, F., Carvalho, I., Góes, F., Pardo, T., and Benevenuto, F. (2022a). Contextualaware and expert data resources for brazilian portuguese hate speech detection.
Vargas, F., Carvalho, I., Rodrigues de Góes, F., Pardo, T., and Benevenuto, F. (2022b). HateBR: A large expert annotated corpus of Brazilian Instagram comments for offensive language and hate speech detection. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7174–7183, Marseille, France. European Language Resources Association.
Waseem, Z. (2016). Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter. In Bamman, D., Doğruöz, A. S., Eisenstein, J., Hovy, D., Jurgens, D., O’Connor, B., Oh, A., Tsur, O., and Volkova, S., editors, Proceedings of the First Workshop on NLP and Computational Social Science, pages 138–142, Austin, Texas. Association for Computational Linguistics.
Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., and Kumar, R. (2019). SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Zhou, D., Bousquet, O., Lal, T. N., Weston, J., and Schölkopf, B. (2004). Learning with local and global consistency. In Advances in neural information processing systems, pages 321–328, MA, USA.
Zhu, X., Ghahramani, Z., and Lafferty, J. (2003). Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML’03, page 912–919. AAAI Press.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1877–1901, Online. Curran Associates, Inc.
Costa Bertaglia, T. F. and Volpe Nunes, M. d. G. (2016). Exploring word embeddings for unsupervised textual user-generated content normalization. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT), pages 112–120, Osaka, Japan. The COLING 2016 Organizing Committee.
de Pelle, R. and Moreira, V. (2017). Offensive comments in the brazilian web: a dataset and baseline results. In VI Brazilian Workshop on Social Network Analysis and Mining, pages 510–519, São Paulo, Brazil. SBC.
Fortuna, P., Rocha da Silva, J., Soler-Company, J., Wanner, L., and Nunes, S. (2019). A hierarchically-labeled Portuguese hate speech dataset. In Roberts, S. T., Tetreault, J., Prabhakaran, V., and Waseem, Z., editors, Proceedings of the Third Workshop on Abusive Language Online, pages 94–104, Florence, Italy. Association for Computational Linguistics.
Hartmann, N., Fonseca, E., Shulby, C., Treviso, M., Silva, J., and Aluísio, S. (2017). Portuguese word embeddings: Evaluating on word analogies and natural language tasks. In Proceedings of the 11th Brazilian Symposium in Information and Human Language Technology, pages 122–131, Uberlândia, Brazil. Sociedade Brasileira de Computação.
Hettiachchi, D., Holcombe-James, I., Livingstone, S., de Silva, A., Lease, M., Salim, F., and Sanderson, M. (2023). How crowd worker factors influence subjective annotations: A study of tagging misogynistic hate speech in tweets. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 11:38–50.
Lees, A., Tran, V. Q., Tay, Y., Sorensen, J., Gupta, J., Metzler, D., and Vasserman, L. (2022). A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, page 3197–3207, New York, NY, USA. Association for Computing Machinery.
Leite, J. A., Silva, D., Bontcheva, K., and Scarton, C. (2020). Toxic language detection in social media for Brazilian Portuguese: New dataset and multilingual analysis. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 914–924, Suzhou, China. Association for Computational Linguistics.
Nascimento, F. R., Cavalcanti, G. D., and Da Costa-Abreu, M. (2022). Unintended bias evaluation: An analysis of hate speech detection and gender bias mitigation on social media using ensemble learning. Expert Systems with Applications, 201:117032.
Oliveira, A., Cecote, T., Silva, P., Castro Gertrudes, J., Freitas, V., and Luz, E. (2023). How good is chatgpt for detecting hate speech in portuguese? pages 94–103.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830.
Ross, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., and Wojatzki, M. (2016). Measuring the reliability of hate speech annotations: The case of the european refugee crisis. In Dipper, S., editor, NLP4CMC III: 3rd Workshop on Natural Language Processing for Computer-Mediated Communication, Bochumer Linguistische Arbeitsberichte, pages 6–9, Germany. Ruhr-Universitat Bochum.
Rossi, R. G. (2015). Classificação automática de textos por meio de aprendizado de máquina baseado em redes. PhD thesis, Instituto de Ciências Matemáticas e de Computação.
Russell, S. J. and Norvig, P. (2009). Artificial Intelligence: a modern approach. Pearson, 3 edition.
Saraiva, G. D., Anchiêta, R., Neto, F. A. R., and Moura, R. (2021). A semi-supervised approach to detect toxic comments. In Mitkov, R. and Angelova, G., editors, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1261–1267, Held Online. INCOMA Ltd.
Souza, F., Nogueira, R., and Lotufo, R. (2020). Bertimbau: Pretrained bert models for brazilian portuguese. In Cerri, R. and Prati, R. C., editors, Proceedings of the 9th Brazilian Conference on Intelligent Systems, pages 403–417, Rio Grande. Springer International Publishing.
Vargas, F., Carvalho, I., Góes, F., Pardo, T., and Benevenuto, F. (2022a). Contextualaware and expert data resources for brazilian portuguese hate speech detection.
Vargas, F., Carvalho, I., Rodrigues de Góes, F., Pardo, T., and Benevenuto, F. (2022b). HateBR: A large expert annotated corpus of Brazilian Instagram comments for offensive language and hate speech detection. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7174–7183, Marseille, France. European Language Resources Association.
Waseem, Z. (2016). Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter. In Bamman, D., Doğruöz, A. S., Eisenstein, J., Hovy, D., Jurgens, D., O’Connor, B., Oh, A., Tsur, O., and Volkova, S., editors, Proceedings of the First Workshop on NLP and Computational Social Science, pages 138–142, Austin, Texas. Association for Computational Linguistics.
Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., and Kumar, R. (2019). SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Zhou, D., Bousquet, O., Lal, T. N., Weston, J., and Schölkopf, B. (2004). Learning with local and global consistency. In Advances in neural information processing systems, pages 321–328, MA, USA.
Zhu, X., Ghahramani, Z., and Lafferty, J. (2003). Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML’03, page 912–919. AAAI Press.
Publicado
21/07/2024
Como Citar
R. NETO, Francisco A.; ANCHIÊTA, Rafael T.; MOURA, Raimundo S.; SANTANA, André M..
Abordagem Semi-Supervisionada para Anotação de Linguagem Tóxica. In: BRAZILIAN WORKSHOP ON SOCIAL NETWORK ANALYSIS AND MINING (BRASNAM), 13. , 2024, Brasília/DF.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 116-129.
ISSN 2595-6094.
DOI: https://doi.org/10.5753/brasnam.2024.2965.
