A Systematic Analysis of Methods for Uncertainty Quantification in Distribution-based Recommender Systems

• Joel Machado Pires Universidade Federal da Bahia (UFBA) https://orcid.org/0000-0002-8428-3516
• Eduardo Ferreira da Silva Universidade Federal da Bahia (UFBA) https://orcid.org/0000-0001-8911-5805
• Frederico Araújo Durão Universidade Federal da Bahia (UFBA)

DOI: https://doi.org/10.5753/sbbd.2026.249287

Resumo

Recommender systems (RS) have become necessary tools for personalizing content and services across digital platforms, yet their confidence estimation remains a critical concern. Confidence estimation in RS aims to answer how confident the model is in its predictions. Despite the advances, integrating confidence into RS models often degrades performance, with outcomes strongly dependent on both dataset characteristics and architectural suitability. To investigate these issues, we conducted a comprehensive empirical study over established distribution-based confidence-aware models, OrdRec, CPMF, CBPMF, LBD, PRGAT, and PRLIGHTGCN. We further performed systematic capacity ablations on deeper models. The methodology combined expected calibration error, confidence-error correlation, rating distribution comparisons, and capacity-vs-performance ablation. Our results show that integrating confidence estimation does not consistently improve and can degrade point-estimate performance in recommendation models. Increasing model capacity fails to yield gains and can negatively impact performance, even in the training set, indicating optimization and data constraints. Moreover, calibration quality is not reliably aligned with predictive accuracy. These models may exhibit meaningful confidence–error correlation while remaining poorly calibrated in absolute terms.

Palavras-chave: confidence, calibration, recommender systems

Referências

Chen, Q., Li, X., Fang, Y., and Wang, M. (2025). Advancing confidence calibration and quantification in medication recommendation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, KDD ’25, page 106–117, New York, NY, USA. Association for Computing Machinery.

Coscrato, V. and Bridge, D. (2023). Estimating and evaluating the uncertainty of rating predictions and top-n recommendations in recommender systems. ACM Trans. Recomm. Syst., 1(2).

da Silva, D., Pires, J., and Durão, F. (2025). Exploiting surrogate submodular and cost-effective lazy forward algorithms for calibrated recommendations. In Anais do XL Simpósio Brasileiro de Bancos de Dados, pages 98–111, Porto Alegre, RS, Brasil. SBC.

de Lourdes M. Silva, M., Chaves, I., Mendonça, A. L., Neto, E. D., and Machado, J. (2025). Twix: Balancing fairness and utility in item exposure for recommendation systems. In Anais do XL Simpósio Brasileiro de Bancos de Dados, pages 427–440, Porto Alegre, RS, Brasil. SBC.

dos Santos, J. V. F., Nascimento, R., Camargo, A. C., Canuto, S., Costa, G., and Sousa, D. (2025). Nova base de dados brasileira para sistemas de recomendação de artigos científicos. In Anais do XL Simpósio Brasileiro de Bancos de Dados, pages 289–302, Porto Alegre, RS, Brasil. SBC.

Goldberg, K., Roeder, T., Gupta, D., and Perkins, C. (2001). Eigentaste: A constant time collaborative filtering algorithm. information retrieval, 4(2):133–151.

Harper, F. M. and Konstan, J. A. (2015). The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 5(4).

He, R. and McAuley, J. (2016). Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web, WWW ’16, page 507–517, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.

Knyazev, N. and Oosterhuis, H. (2023). A lightweight method for modeling confidence in recommendations with learned beta distributions. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 306–317.

Koren, Y. and Sill, J. (2011). Ordrec: an ordinal model for predicting personalized item rating distributions. In Proceedings of the Fifth ACM Conference on Recommender Systems, RecSys ’11, page 117–124, New York, NY, USA. Association for Computing Machinery.

Naeini, M. P., Cooper, G., and Hauskrecht, M. (2015). Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29.

Negrão, A., Rocha, G., dos Santos, L., Malaquias, P., Pedrosa, R., Fortes, R., and Silva, P. (2025). Mitigando impactos de distribuições não-iid em aprendizagem federada para sistemas de recomendação. In Anais do XL Simpósio Brasileiro de Bancos de Dados, pages 413–426, Porto Alegre, RS, Brasil. SBC.

Pires, J. M., Silva, E. F. d., and Durão, F. A. (2026). Exploiting distribution-based confidence integration in graph neural network recommenders. Applied Intelligence, 56(5):142.

Wang, C., Liu, Q., Wu, R., Chen, E., Liu, C., Huang, X., and Huang, Z. (2018). Confidence-aware matrix factorization for recommender systems. In Proceedings of the AAAI Conference on artificial intelligence, volume 32.

Wu, L. (2024). Towards trustworthy graph neural networks and their applications in recommender systems. In 2024 IEEE International Conference on Big Data (BigData), pages 8250–8252.

Xue, H.-J., Dai, X., Zhang, J., Huang, S., and Chen, J. (2017). Deep matrix factorization models for recommender systems. In Ijcai, volume 17, pages 3203–3209. Melbourne, Australia.