Ensemble Learning through Rashomon Sets
Abstract
Creating models from previous observations and ensuring effectiveness on new data is the essence of machine learning. However, selecting models that generalize well to future data remains a challenging task. In this work, we investigate how models perform across datasets with distinct underlying data generation functions but constitute co-related tasks. The key motivation is to study the Rashomon Effect, which appears whenever the learning problem admits a set of models that all perform roughly equally well. Real-world problems often exhibit multiple local structures in data space, leading to a non-convex error surface and multiple high-performing models which literature suggests to be subject to the Rashomon Effect. Our approach is to stratify, during training, the solution space into model groups that are either coherent or contrasting given both performance and explanations. From these Rashomon groups, we build an ensemble ensuring that each constituent covers a distinct region of the solution space. We validate our approach by performing a series of experiments in both open and closed real-world datasets. Our method outperforms state-of-the-art techniques, improving AUROC by up to 0.20+ when the Rashomon ratio is large.
References
Breiman, L. (2001). Statistical modeling: the two cultures. Statistical Science. A Review Journal of the Institute of Mathematical Statistics, 16(3):199–231.
Chen, C., Yuan, J., Lu, Y., Liu, Y., Su, H., Yuan, S., and Liu, S. (2021). Oodanalyzer: Interactive analysis of out-of-distribution samples. IEEE Trans. Vis. Comput. Graph., 27(7):3335–3349.
Endres, D. M. and Schindelin, J. E. (2003). A new metric for probability distributions. IEEE Transactions on Information theory, 49(7).
Fisher, A., Rudin, C., and Dominici, F. (2019). All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously. J. Mach. Learn. Res., 20:177:1–177:81.
Hinns, J., Fan, X., Liu, S., Raghava Reddy Kovvuri, V., Yalcin, M. O., and Roggenbach, M. (2021). An initial study of machine learning underspecification using feature attribution explainable ai algorithms: A covid-19 virus transmission case study. In Pacific Rim International Conference on Artificial Intelligence. Springer.
Lundberg, S. and Lee, S. (2017). A unified approach to interpreting model predictions. In Annual Conf. on Neural Information Processing Systems, pages 4768–4777.
Madras, D., Atwood, J., and D’Amour, A. (2020). Detecting extrapolation with local ensembles. In International Conference on Learning Representations.
Semenova, L. and Rudin, C. (2019). A study in rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning. CoRR, abs/1908.01755.
Shmueli, G. (2010). To explain or to predict? Statistical Science, 25(3):289–310.
Sun, T., Zanocco, C. M., Zuin, G., Buechler, R., Flora, J. A., Galuppo, F., Soto, H., Veloso, A., and Rajagopal, R. (2023). Assessing climate change impacts on electricity consumption: Micro to macro perspectives from california’s commercial sector to brazil’s national grid. AGU23.
Veloso, A., Caramelli, P., Borges, K. B. G., Araújo, D., Zuin, G., Alves, T. H., and Ziviani, N. (2023). Processo centrado no humano para elaboração de modelos baseados em aprendizado de máquina e usos. Brazil patent BR 102021015411-0 A8.
Zuin, G. (2023). Ensemble Learning through Rashomon Sets. PhD thesis, Universidade Federal de Minas Gerais.
Zuin, G., Araujo, D., Ribeiro, V., Seiler, M. G., Prieto, W. H., Pintão, M. C., dos Santos Lazari, C., Granato, C. F. H., and Veloso, A. (2022a). Prediction of sars-cov-2-positivity from million-scale complete blood counts using machine learning. Communications medicine, 2(1):1–12.
Zuin, G., Buechler, R., Sun, T., Zanocco, C., Castro, D., Veloso, A., and Rajagopal, R. (2022b). Revealing the impact of extreme events on electricity consumption in brazil: A data-driven counterfactual approach. In 2022 IEEE Power & Energy Society General Meeting (PESGM), pages 1–5.
Zuin, G., Buechler, R., Sun, T., Zanocco, C., Galuppo, F., Veloso, A., and Rajagopal, R. (2023a). Extreme event counterfactual analysis of electricity consumption in brazil: Historical impacts and future outlook under climate change. Energy, page 128101.
Zuin, G., Chaimowicz, L., and Veloso, A. (2020). Deep learning techniques for explainable resource scales in collectible card games. IEEE Transactions on Games, pages 1–1.
Zuin, G., Marcelino, F., Borges, L., Couto, J., Jorge, V., Laurindo, M., Barcelos, G., Cunha, M., Alvarenga, V., Rodrigues, H., et al. (2021). Predicting heating sliver in duplex stainless steels manufacturing through rashomon sets. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.
Zuin, G., Parreiras, L., Melo, L., Barros, G., Lomeu, H., Melo, B., Marini, W., Lott, D., and De Souza, M. (2023b). An ensemble approach for inconsistency detection in medical bills: A case study. In 2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS), pages 573–578. IEEE.
Zuin, G. and Veloso, A. (2019). Learning a resource scale for collectible card games. In 2019 IEEE Conference on Games (CoG), pages 1–8.
Zuin, G., Veloso, A., Portinari, J. C., and Ziviani, N. (2020). Automatic tag recommendation for painting artworks using diachronic descriptions. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE.
