Automatic label error detection on text datasets labeled with data programming - Leal, Nalbert G. M.; Araújo, Daniel S. A.; Menezes Neto, Elias J. (Anais do Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), 2025)
Supervised machine learning relies on the availability of large volumes of accurately annotated data, a requirement that data programming (DP) alleviates by aggregating weak supervision sources into probabilistic labels. However, DP generated labels remain susceptible to noise, compromising downstream model performance. In this work, we integrate and evaluate four Automatic Error Detection (AED) techniques: Retag, Confident Learning, Source-aware Influence Functions, and Unsupervised Label Functions Correction (ULF); within standard two-stage DP pipelines. Using the WRENCH benchmark on two text-classification tasks (YouTube and SMS spam detection), we optimize each pipeline via Bayesian optimization and collect the metrics Matthews correlation coefficient, accuracy, F1 score, and computational cost. Our experiments show that Influence Functions combined with the Hyper Label Model achieve the most positive trade-off between accuracy improvement and runtime on balanced data, while simpler DP baselines outperform all analyzed AED methods under class imbalance. These findings underscore both the promise and practical limitations of AED in refining weakly supervised workflows, guiding future development of cost-effective label-noise mitigation strategies.