When Annotators Disagree: A Controlled Evaluation of Gender Bias in Sentiment Analysis Using Synthetic Datasets - Carneiro, Érica; Feitosa, Alexander; Guedes, Gustavo (Anais do Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL); 2025: Anais do XVI Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana, 2025)
This study investigates gender-related annotation bias in sentiment classification. First, we introduce a controlled synthetic dataset generation method that simulates parallel male and female sentiment labels with adjustable inter-annotator agreement. Then, we present the Gender Comparison Methodology, which trains classifiers separately on gender-partitioned labels and evaluates their predictions using shared textual inputs. Agreement is assessed using metrics such as Cohen’s Kappa, chi-square test, and Cramér’s V. Results show that even moderate disagreement between annotators leads to systematic model divergence, highlighting the importance of annotator identity in shaping classification behavior and informing fairness-aware auditing practices.