Beyond Random Sampling: Instance Quality-Based Data Partitioning via Item Response Theory

• Lucas Cardoso UFPA / ITV
• Vitor Santos UFPA
• Ricardo Prudêncio UFPE
• José Ribeiro IFPA
• Regiane Kawasaki UFPA
• Ronnie Alves UFPA / ITV

DOI: https://doi.org/10.5753/eniac.2025.13812

Resumo

Robust validation of Machine Learning (ML) models is essential, but traditional data partitioning approaches often ignore the intrinsic quality of each instance. This study proposes the use of Item Response Theory (IRT) parameters to characterize and guide the partitioning of datasets in the model validation stage. The impact of IRT-informed partitioning strategies on the performance of several ML models in four tabular datasets was evaluated. The results obtained demonstrate that IRT reveals an inherent heterogeneity of the instances and highlights the existence of informative subgroups of instances within the same dataset. Based on IRT, balanced partitions were created that consistently help to better understand the tradeoff between bias and variance of the models. In addition, the guessing parameter proved to be a determining factor: training with high-guessing instances can significantly impair model performance and resulted in cases with accuracy below 50%, while other partitions reached more than 70% in the same dataset.

Referências

Aggarwal, C. C. (2017). An introduction to outlier analysis. Springer.

Araujo Santos, V. C., Cardoso, L., and Alves, R. (2023). The quest for the reliability of machine learning models in binary classification on tabular data. Scientific Reports, 13(1):18464.

Baker, F. B. (2001). The basics of item response theory. ERIC.

Cardoso, L. F., Ribeiro Filho, J. d. S., Santos, V. C., Kawasaki Francês, R. S., and Alves, R. C. (2024). Standing on the shoulders of giants. In Brazilian Conference on Intelligent Systems, pages 416–430. Springer.

Cardoso, L. F., Santos, V. C., Francês, R. S. K., Prudêncio, R. B., and Alves, R. C. (2020). Decoding machine learning benchmarks. In Brazilian Conference on Intelligent Systems, pages 412–425. Springer.

de Andrade, D. F., Tavares, H. R., and da Cunha Valle, R. (2000). Teoria da resposta ao item: conceitos e aplicações. ABE, Sao Paulo.

de Sousa Ribeiro Filho, J., Cardoso, L. F. F., da Silva, R. L. S., Carneiro, N. J. S., Santos, V. C. A., and de Oliveira Alves, R. C. (2024). Explanations based on item response theory (exirt): A model-specific method to explain tree-ensemble model in trust perspective. Expert Systems with Applications, 244:122986.

Fan, W. and Bifet, A. (2013). Mining big data: current status, and forecast to the future. ACM SIGKDD explorations newsletter, 14(2):1–5.

He, H. and Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9):1263–1284.

INEP (2012). NOTA TÉCNICA: Teoria de Resposta ao Item.

Jordan, M. I. and Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245):255–260.

Martínez-Plumed, F., Prudêncio, R. B., Martínez-Usó, A., and Hernández-Orallo, J. (2019). Item response theory in ai: Analysing machine learning classifiers at the instance level. Artificial Intelligence, 271:18–42.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830.

Prudêncio, R. B., Hernández-Orallo, J., and Martınez-Usó, A. (2015). Analysis of instance hardness in machine learning using item response theory. In Second International Workshop on Learning over Multiple Contexts in ECML 2015. Porto, Portugal, 11 September 2015, volume 1.

Rizopoulos, D. (2006). ltm: An r package for latent variable modeling and item response theory analyses. Journal of statistical software, 17(5):1–25.

Song, H. and Flach, P. (2021). Efficient and robust model benchmarks with item response theory and adaptive testing.

Vanschoren, J., Van Rijn, J. N., Bischl, B., and Torgo, L. (2014). Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49–60.