Measuring the Degree of Divergence when Labeling Tweets in the Electoral Scenario

• Jéssica S. Santos UFF http://orcid.org/0000-0001-5082-4583
• Flávia Bernardini UFF
• Aline Paes UFF

DOI: https://doi.org/10.5753/brasnam.2021.16131

Resumo

Analyzing electoral trends in political scenarios using social media with data mining techniques has become popular in recent years. A problem in this field is to reliably annotate data during the short period of electoral campaigns. In this paper, we present a methodology to measure labeling divergence and an exploratory analysis of data related to the 2018 Brazilian Presidential Elections. As a result, we point out some of the main characteristics that lead to a high level of divergence during the annotation process in this domain. Our analysis shows a high degree of divergence mainly in regard to sentiment labels. Also, a significant difference was identified between labels obtained by manual annotation and labels obtained using an automatic annotation approach.

Palavras-chave: labeling divergence, electoral domain, sentiment analysis, manual annotation, offensive speech detection

Referências

Bhowmick, P. K., Basu, A., and Mitra, P. (2008). An agreement measure for determining inter-annotator reliability of human judgements on affective text. In Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics, pages 58–65.

Bilal, M., Gani, A., Marjani, M., and Malik, N. (2019). Predicting elections: Social media data and techniques. In 2019 international conference on engineering and emerging technologies (ICEET), pages 1–6. IEEE.

Bobicev, V. and Sokolova, M. (2017). Inter-annotator agreement in sentiment analysis: Machine learning perspective. In RANLP, volume 97.

Burnap, P., Gibson, R., Sloan, L., Southern, R., and Williams, M. (2016). 140 characters to victory?: Using twitter to predict the uk 2015 general election. Electoral Studies, 41:230–233.

Calais Guerra, P. H., Veloso, A., Meira Jr, W., and Almeida, V. (2011). From bias to opinion: a transfer-learning approach to real-time sentiment analysis. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 150–158. ACM.

Chapman, L., Resch, B., Sadler, J., Zimmer, S., Roberts, H., and Petutschnig, A. (2018). Investigating the emotional responses of individuals to urban green space using twitter data: A critical comparison of three different methods of sentiment analysis. Urban Planning, 3(1):21–33.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.

dos Santos, J. S., Paes, A., and Bernardini, F. (2019). Combining labeled datasets for sentiment analysis from different domains based on dataset similarity to predict electors sentiment. In 2019 8th Brazilian Conference on Intelligent Systems (BRACIS), pages 455–460. IEEE.

Dwi Prasetyo, N. and Hauff, C. (2015). Twitter-based election prediction in the developing world. In Proceedings of the 26th ACM Conference on Hypertext & Social Media, pages 149–158. ACM.

Elsahar, H. and Galle, M. (2019). To annotate or not? predicting performance drop under domain shift. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2163–2173.

Fleiss, J. L., Levin, B., Paik, M. C., et al. (1981). The measurement of interrater agreement. Statistical methods for rates and proportions, 2(212-236):22–23.

Gohil, L. and Patel, D. (2019). A sentiment analysis of gujarati text using gujarati senti word net. International Journal of Innovative Technology and Exploring Engineering (IJITEE).

Heredia, B., Prusa, J., and Khoshgoftaar, T. (2017). Exploring the effectiveness of twitter at polling the united states 2016 presidential election. In Collaboration and Internet Computing (CIC), 2017 IEEE 3rd International Conference on, pages 283–290. IEEE.

Huberty, M. (2015). Can we vote with our tweet? on the perennial difficulty of election forecasting with social media. International Journal of Forecasting, 31(3):992–1007.

Karami, A., Bennett, L. S., and He, X. (2018). Mining public opinion about economic issues: Twitter and the us presidential election. International Journal of Strategic Decision Sciences (IJSDS), 9(1):18–28.

Krippendorff, K. (2004). Content analysis – an introduction to its methodology. Beverly Hills, CA: Sage Publications.

Krippendorff, K. (2011). Computing krippendorff’s alpha-reliability.

Liu, B. (2020). Sentiment Analysis: Mining Opinions, Sentiments, and Emotions. Studies in Natural Language Processing. Cambridge University Press, 2 edition.

Mahendiran, A., Wang, W., Lira, J. A. S., Huang, B., Getoor, L., Mares, D., and Ramakrishnan, N. (2014). Discovering evolving political vocabulary in social media. In 2014 International Conference on Behavioral, Economic, and Socio-Cultural Computing (BESC2014), pages 1–7. IEEE.

Okeowo, A. (2017). Hate on the rise after trump’s election. The New Yorker, 17.

Shannon, C. E. (2001). A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3–55.

Teruel, M., Cardellino, C., Cardellino, F., Alemany, L. A., and Villata, S. (2018). Increasing argument annotation reproducibility by using inter-annotator agreement to improve guidelines. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).

Tsakalidis, A., Papadopoulos, S., Cristea, A. I., and Kompatsiaris, Y. (2015). Predicting elections for multiple countries using twitter and polls. IEEE Intelligent Systems, 30(2):10–17.

Unankard, S., Li, X., Sharaf, M., Zhong, J., and Li, X. (2014). Predicting elections from social networks based on sub-event detection and sentiment analysis. In International Conference on Web Information Systems Engineering, pages 1–16. Springer.

Wu, F., Huang, Y., and Yuan, Z. (2017). Domain-specific sentiment classification via fusing sentiment knowledge from multiple sources. Information Fusion, 35:26–37.