Evaluating Instance Selection for Active Learning using Label-Agnostic Instance Hardness Measures

• Bárbara de Morais Coutinho ITA
• Patricia S. M. Ueda ITA
• Ana Carolina Lorena ITA

DOI: https://doi.org/10.5753/eniac.2026.31141

Resumo

Supervised machine learning often requires large amounts of labeled data. Active Learning (AL) addresses this limitation by iteratively selecting the most informative instances from an unlabeled pool, aiming to maximize predictive performance. Although traditional AL strategies are largely driven by model uncertainty, alternative approaches exploit Instance Hardness Metrics (IHMs) to prioritize challenging samples. However, many IHMs rely on true class labels for query instances, which are unavailable during the AL selection. This study investigates the impact of replacing traditional, label-dependent hardness metrics with adjusted, label-agnostic versions within an AL framework. Experiments were conducted on 18 diverse datasets using an XGBoost classifier, and performance was assessed through the Area Under the Kappa Curve. The adjusted metrics eliminate the need for prior label information for query instances, making them considerably more practical for real-world AL applications. Among the evaluated methods, the adjusted k-Disagreeing Neighbors metric consistently achieved the strongest performance across most datasets.

Referências

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46.

Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan):1–30.

Lewis, D. D. and Gale, W. A. (1994). A sequential algorithm for training text classifiers. In SIGIR’94, pages 3–12.

Lorena, A. C., Paiva, P. Y., and Prudêncio, R. B. (2024). Trusting my predictions: on the value of instance-level analysis. ACM Computing Surveys, 56(7):1–28.

Monarch, R. M. (2021). Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster.

Nogueira, G. S. C. d., P. dos Santos, D., CP L F. de Carvalho, A., and PF Garcia, L. (2025). Hardness sampling: Exploring instance hardness in pool-based active learning. In Brazilian Conference on Intelligent Systems, pages 3–17. Springer.

Quinlan, J. R. (1986). Induction of decision trees. Machine learning, 1(1):81–106.

Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., and Wang, X. (2021). A survey of deep active learning. ACM computing surveys (CSUR), 54(9):1–40.

Scheffer, T., Decomain, C., and Wrobel, S. (2001). Active hidden markov models for information extraction. In Proceedings of the International Conference on Advances in Intelligent Data Analysis, IDA ’01, pages 309–318, Berlin, Heidelberg. Springer-Verlag.

Settles, B. (2009). Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison.

Shwartz-Ziv, R. and Armon, A. (2022). Tabular data: Deep learning is not all you need. Information fusion, 81:84–90.

Smith, M. R., Martinez, T., and Giraud-Carrier, C. (2014). An instance level analysis of data complexity. Machine learning, 95(2):225–256.

Ueda, P. S., Rivolli, A., and Lorena, A. C. (2024). An instance level analysis of classification difficulty for unlabeled data. In Brazilian Conference on Intelligent Systems, pages 141–155. Springer.