Time-aware Confidence Calibration Analysis in Recommender Systems Methods
Resumo
Recommender Systems (RS) are typically evaluated using accuracy and ranking metrics, but these metrics do not show whether the distribution of predicted ratings resembles the distribution of observed ratings. This aspect is especially important in confidence-aware recommendation, where confidence calibration indicates whether predictions assigned higher confidence are indeed more reliable. This work investigates the relationship among distributional fidelity, predictive performance, and confidence calibration in RS under temporal recommendation settings.We evaluate CPMF, LBD, OrdRec, PRLIGHTGCN, and MC Dropout on the MovieLens 1M dataset using an incremental time-series cross-validation protocol, and analyze the effects ofWasserstein-based distribution regularization. The results show that models produce characteristic prediction distributions that remain relatively stable over time but do not converge toward the empirical rating distribution, even as additional interactions become available. Moreover, strong recommendation accuracy does not necessarily imply statistical fidelity to the observed data. Confidence calibration evolves over time and depends strongly on model architecture, while modifications to the global prediction distribution have limited effects on user-item confidence calibration. Overall, distributional fidelity emerges as a complementary dimension for evaluating trustworthy and confidence-aware recommender systems.
Palavras-chave:
confidence, distribution, calibration, recommender systems
Referências
Charu C. Aggarwal. 2016. Recommender Systems: The Textbook (1st ed.). Springer Publishing Company, Incorporated, Online.
Gabrielle Alves and Marcelo Manzato. 2024. Furando a Bolha: Nudges Digitais em Sistemas de Recomendação. In Anais Estendidos do XXX Simpósio Brasileiro de Sistemas Multimídia e Web (Juiz de Fora/MG). SBC, Porto Alegre, RS, Brasil, 11–12. DOI: 10.5753/webmedia_estendido.2024.244374
Shun-ichi Amari, Ryo Karakida, and Masafumi Oizumi. 2018. Information geometry connecting Wasserstein distance and Kullback–Leibler divergence via the entropy-relaxed transportation problem. Information Geometry 1, 1 (2018), 13–37.
Qianyu Chen, Xin Li, Yujie Fang, and Mingzhong Wang. 2025. Advancing Confidence Calibration and Quantification in Medication Recommendation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (Toronto ON, Canada) (KDD ’25). Association for Computing Machinery, New York, NY, USA, 106–117. DOI: 10.1145/3690624.3709232
Victor Coscrato and Derek Bridge. 2023. Estimating and Evaluating the Uncertainty of Rating Predictions and Top-n Recommendations in Recommender Systems. ACM Trans. Recomm. Syst. 1, 2, Article 7 (apr 2023), 34 pages. DOI: 10.1145/3584021
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 1050–1059. [link]
F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst. 5, 4, Article 19 (Dec. 2015), 19 pages. DOI: 10.1145/2827872
Norman Knyazev and Harrie Oosterhuis. 2023. A lightweight method for modeling confidence in recommendations with learned beta distributions. In Proceedings of the 17th ACM Conference on Recommender Systems. 306–317. DOI: 10.1145/3604915.3608788
Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix Factorization Techniques for Recommender Systems. Computer 42, 8 (2009), 30–37. DOI: 10.1109/MC.2009.263
Yehuda Koren and Joe Sill. 2011. OrdRec: an ordinal model for predicting personalized item rating distributions. In Proceedings of the Fifth ACM Conference on Recommender Systems (Chicago, Illinois, USA) (RecSys ’11). Association for Computing Machinery, New York, NY, USA, 117–124. DOI: 10.1145/2043932.2043956
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 29. DOI: 10.1609/aaai.v29i1.9602
Joel Machado Pires, Eduardo Ferreira da Silva, and Frederico Araújo Durão. 2026. Exploiting distribution-based confidence integration in graph neural network recommenders. Applied Intelligence 56, 5 (2026), 142. DOI: 10.1007/s10489-026-07187-0
Joseph Lee Rodgers and W. Alan Nicewander. 1988. Thirteen Ways to Look at the Correlation Coefficient. The American Statistician 42, 1 (1988), 59–66. DOI: 10.1080/00031305.1988.10475524
Eduardo Silva, Joel Pires, Denis Dantas, Mayki Oliveira, and Frederico Durão. 2025. A Gated Review Attention Framework for Topics in Graph-Based Recommenders. In Proceedings of the 31st Brazilian Symposium on Multimedia and the Web (Rio de Janeiro/RJ). SBC, Porto Alegre, RS, Brasil, 19–27. DOI: 10.5753/webmedia.2025.15515
Rodrigo Souza and Marcelo Manzato. 2024. Explorando Formas de Calibração e Redução do Viés de Popularidade em Sistemas de Recomendação. In Anais Estendidos do XXX Simpósio Brasileiro de Sistemas Multimídia e Web (Juiz de Fora/MG). SBC, Porto Alegre, RS, Brasil, 9–10. DOI: 10.5753/webmedia_estendido.2024.244380
Chao Wang, Qi Liu, Runze Wu, Enhong Chen, Chuanren Liu, Xunpeng Huang, and Zhenya Huang. 2018. Confidence-aware matrix factorization for recommender systems. In Proceedings of the AAAI Conference on artificial intelligence, Vol. 32. DOI: 10.1609/aaai.v32i1.11251
Longfeng Wu. 2024. Towards Trustworthy Graph Neural Networks and Their Applications in Recommender Systems. In 2024 IEEE International Conference on Big Data (BigData). 8250–8252. DOI: 10.1109/BigData62323.2024.10825262
Eva Zangerle and Christine Bauer. 2022. Evaluating Recommender Systems: Survey and Framework. ACM Comput. Surv. 55, 8, Article 170 (Dec. 2022), 38 pages. DOI: 10.1145/3556536
André Zanon, Leonardo Rocha, and Marcelo Manzato. 2025. An exploration of recommender systems explanation paradigms: generating and evaluating syntactic, semantic, and generative models with knowledge graphs: An Extended Abstract. In Anais Estendidos do XXXI Simpósio Brasileiro de Sistemas Multimídia e Web (Rio de Janeiro/RJ). SBC, Porto Alegre, RS, Brasil, 19–20. DOI: 10.5753/webmedia_estendido.2025.16351
Gabrielle Alves and Marcelo Manzato. 2024. Furando a Bolha: Nudges Digitais em Sistemas de Recomendação. In Anais Estendidos do XXX Simpósio Brasileiro de Sistemas Multimídia e Web (Juiz de Fora/MG). SBC, Porto Alegre, RS, Brasil, 11–12. DOI: 10.5753/webmedia_estendido.2024.244374
Shun-ichi Amari, Ryo Karakida, and Masafumi Oizumi. 2018. Information geometry connecting Wasserstein distance and Kullback–Leibler divergence via the entropy-relaxed transportation problem. Information Geometry 1, 1 (2018), 13–37.
Qianyu Chen, Xin Li, Yujie Fang, and Mingzhong Wang. 2025. Advancing Confidence Calibration and Quantification in Medication Recommendation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1 (Toronto ON, Canada) (KDD ’25). Association for Computing Machinery, New York, NY, USA, 106–117. DOI: 10.1145/3690624.3709232
Victor Coscrato and Derek Bridge. 2023. Estimating and Evaluating the Uncertainty of Rating Predictions and Top-n Recommendations in Recommender Systems. ACM Trans. Recomm. Syst. 1, 2, Article 7 (apr 2023), 34 pages. DOI: 10.1145/3584021
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of The 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 48), Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 1050–1059. [link]
F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst. 5, 4, Article 19 (Dec. 2015), 19 pages. DOI: 10.1145/2827872
Norman Knyazev and Harrie Oosterhuis. 2023. A lightweight method for modeling confidence in recommendations with learned beta distributions. In Proceedings of the 17th ACM Conference on Recommender Systems. 306–317. DOI: 10.1145/3604915.3608788
Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix Factorization Techniques for Recommender Systems. Computer 42, 8 (2009), 30–37. DOI: 10.1109/MC.2009.263
Yehuda Koren and Joe Sill. 2011. OrdRec: an ordinal model for predicting personalized item rating distributions. In Proceedings of the Fifth ACM Conference on Recommender Systems (Chicago, Illinois, USA) (RecSys ’11). Association for Computing Machinery, New York, NY, USA, 117–124. DOI: 10.1145/2043932.2043956
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 29. DOI: 10.1609/aaai.v29i1.9602
Joel Machado Pires, Eduardo Ferreira da Silva, and Frederico Araújo Durão. 2026. Exploiting distribution-based confidence integration in graph neural network recommenders. Applied Intelligence 56, 5 (2026), 142. DOI: 10.1007/s10489-026-07187-0
Joseph Lee Rodgers and W. Alan Nicewander. 1988. Thirteen Ways to Look at the Correlation Coefficient. The American Statistician 42, 1 (1988), 59–66. DOI: 10.1080/00031305.1988.10475524
Eduardo Silva, Joel Pires, Denis Dantas, Mayki Oliveira, and Frederico Durão. 2025. A Gated Review Attention Framework for Topics in Graph-Based Recommenders. In Proceedings of the 31st Brazilian Symposium on Multimedia and the Web (Rio de Janeiro/RJ). SBC, Porto Alegre, RS, Brasil, 19–27. DOI: 10.5753/webmedia.2025.15515
Rodrigo Souza and Marcelo Manzato. 2024. Explorando Formas de Calibração e Redução do Viés de Popularidade em Sistemas de Recomendação. In Anais Estendidos do XXX Simpósio Brasileiro de Sistemas Multimídia e Web (Juiz de Fora/MG). SBC, Porto Alegre, RS, Brasil, 9–10. DOI: 10.5753/webmedia_estendido.2024.244380
Chao Wang, Qi Liu, Runze Wu, Enhong Chen, Chuanren Liu, Xunpeng Huang, and Zhenya Huang. 2018. Confidence-aware matrix factorization for recommender systems. In Proceedings of the AAAI Conference on artificial intelligence, Vol. 32. DOI: 10.1609/aaai.v32i1.11251
Longfeng Wu. 2024. Towards Trustworthy Graph Neural Networks and Their Applications in Recommender Systems. In 2024 IEEE International Conference on Big Data (BigData). 8250–8252. DOI: 10.1109/BigData62323.2024.10825262
Eva Zangerle and Christine Bauer. 2022. Evaluating Recommender Systems: Survey and Framework. ACM Comput. Surv. 55, 8, Article 170 (Dec. 2022), 38 pages. DOI: 10.1145/3556536
André Zanon, Leonardo Rocha, and Marcelo Manzato. 2025. An exploration of recommender systems explanation paradigms: generating and evaluating syntactic, semantic, and generative models with knowledge graphs: An Extended Abstract. In Anais Estendidos do XXXI Simpósio Brasileiro de Sistemas Multimídia e Web (Rio de Janeiro/RJ). SBC, Porto Alegre, RS, Brasil, 19–20. DOI: 10.5753/webmedia_estendido.2025.16351
Publicado
27/10/2026
Como Citar
PIRES, Joel Machado; SILVA, Eduardo Ferreira da; OLIVEIRA, Mayki dos Santos; DURÃO, Frederico Araújo; MANZATO, Marcelo G..
Time-aware Confidence Calibration Analysis in Recommender Systems Methods. In: BRAZILIAN SYMPOSIUM ON MULTIMEDIA AND THE WEB (WEBMEDIA), 32. , 2026, Lavras/MG.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 480-488.
DOI: https://doi.org/10.5753/webmedia.2026.30745.
