Why Ignore Content? A Guideline for Intrinsic Evaluation of Item Embeddings for Collaborative Filtering

• Pedro R. Pires UFSCar
• Bruno B. Rizzi BTG Pactual
• Tiago A. Almeida UFSCar

DOI: https://doi.org/10.5753/webmedia.2024.243199

Resumo

With the constant growth in available information and the popularization of technology, recommender systems have to deal with an increasing number of users and items. This leads to two problems in representing items: scalability and sparsity. Therefore, many recommender systems aim to generate low-dimensional dense representations of items. Matrix factorization techniques are popular, but models based on neural embeddings have recently been proposed and are gaining ground in the literature. Their main goal is to learn dense representations with intrinsic meaning. However, most studies proposing embeddings for recommender systems ignore this property and focus only on extrinsic evaluations. This study presents a guideline for assessing the intrinsic quality of matrix factorization and neural-based embedding models for collaborative filtering, comparing the results with a traditional extrinsic evaluation. To enrich the evaluation pipeline, we suggest adapting an intrinsic evaluation task commonly employed in the Natural Language Processing literature, and we propose a novel strategy for evaluating the learned representation compared to a content-based scenario. Finally, every mentioned technique is analyzed over established recommender models, and the results show how vector representations that do not yield good recommendations can still be useful in other tasks that demand intrinsic knowledge, highlighting the potential of this perspective of evaluation.

Palavras-chave: embeddings, intrinsic evaluation, qualitative evaluation, recommender systems, similarity tables, intruder detection, autotagging

Referências

Gediminas Adomavicius and Alexander Tuzhilin. 2005. Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions. IEEE Transactions on Knowledge and Data Engineering 17, 6 (2005), 734–749. DOI: 10.1109/TKDE.2005.99

Oren Barkan and Noam Koenigstein. 2016. Item2Vec: Neural Item Embedding For Collaborative Filtering. In IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP 2016). IEEE, Vietri sul Mare, Italy, 1–6. DOI: 10.1109/MLSP.2016.7738886

Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL ‘14). Association for Computational Linguistics, Baltimore, MD, USA, 238–247. DOI: 10.3115/v1/P14-1023

J. Bobadilla, F. Ortega, A. Hernando, and A. Gutiérrez. 2013. Recommender systems survey. Knowledge-Based Systems 46 (2013), 109–132. DOI: 10.1016/j.knosys.2013.03.012

Hugo Caselles-Duprés, Florian Lesaint, and Jimena Royo-Letelier. 2018. Word2vec applied to recommendation: hyperparameters matter. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ‘18). Association for Computing Machinery, Vancouver, Canada, 352–356. DOI: 10.1145/3240323.3240377

Chao Chang, Junming Zhou, Yu Weng, Xiangwei Zeng, Zhengyang Wu, Chang-Dong Wang, and Yong Tang. 2023. KGTN: Knowledge Graph Transformer Network for explainable multi-category item recommendation. Knowledge-Based Systems 278 (2023), 110854. DOI: 10.1016/j.knosys.2023.110854

Hao Chen, Zefan Wang, Feiran Huang, Xiao Huang, Yue Xu, Yishi Lin, Peng He, and Zhoujun Li. 2022. Generative Adversarial Framework for Cold-Start Item Recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’22). Association for Computing Machinery, Anchorage, AK, USA, 2565–2571. DOI: 10.1145/3477495.3531897

Gabriel de Souza P. Moreira, Dietmar Jannach, and Adilson Marques da Cunha. 2019. On the Importance of News Content Representation in Hybrid Neural Session-based Recommender Systems. IEEE Access 7 (2019), 169185–169203. DOI: 10.1109/ACCESS.2019.2954957

Janez Demšar. 2006. Statistical Comparisons of Classifiers over Multiple Data Sets. The Journal of Machine Learning Research 7 (2006), 1–30. DOI: 10.5555/1248547.1248548

Chengxin Ding, Zhongying Zhao, Chao Li, Yanwei Yu, and Qingtian Zeng. 2023. Session-based recommendation with hypergraph convolutional networks and sequential information embeddings. Expert Systems with Applications 223, 119875 (2023), 1–11. DOI: 10.1016/j.eswa.2023.119875

Douglas Eck, Paul Lamere, Thierry Bertin-Mahieux, and Stephen Green. 2007. Automatic generation of social tags for music recommendation. In Proceedings of the 20th International Conference on Neural Information Processing Systems (NIPS 2007). Curran Associates Inc., Vancouver, Canada, 385–392. DOI: 10.5555/2981562.2981611

Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. 2016. Problems With Evaluation of Word Embeddings Using Word Similarity Tasks. In Proceedings of the 1st Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics, Berlin, Germany, 30–35. DOI: 10.18653/v1/W16-2506

Ralph José Rassweiler Filho, Jônatas Wehrmann, and Rodrigo C. Barros. 2017. Leveraging Deep Visual Features for Content-based Movie Recommender Systems. In Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN 2017). IEEE, Anchorage, AK, USA, 604–611. DOI: 10.1109/IJCNN.2017.7965908

Claudiu S. Firan, Wolfgang Nejdl, and Raluca Paiu. 2007. The Benefit of Using TagBased Profiles. In Proceedings of the 5th Latin American Web Conference (LA-WEB‘07). IEEE Computer Society, Santiago, Chile, 32–41. DOI: 10.1109/LA-WEB.2007.24

Peng FU, Jiang hua LV, Shi long MA, and Bing jie LI. 2017. Attr2vec: A Neural Network Based Item Embedding Method. In Proceedings of the 2nd International Conference on Computer, Mechatronics and Electronic Engineering (CMEE 2017). DEStech Publications, Xiamen, China, 300–307. DOI: 10.12783/dtcse/cmee2017/19993

Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. 2023. Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System. arXiv: 2303.14524 (2023), 1–17. DOI: 10.48550/arXiv.2303.14524

Anna Gladkova and Aleksandr Drozd. 2016. Intrinsic Evaluations of Word Embeddings: What Can We Do Better?. In Proceedings of the 1st Workshop on Evaluating Vector Space Representations for NLP. Association for Computational Linguistics, Berlin, Germany, 36–42. DOI: 10.18653/v1/W16-2507

Mihajlo Grbovic, Vladan Radosavljevic, Nemanja Djuric, Narayan Bhamidipati, Jaikit Savla, Varun Bhagwan, and Doug Sharp. 2015. E-commerce in Your Inbox: Product Recommendations at Scale. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ‘15). Association for Computing Machinery, Sydney, Australia, 1809–1818. DOI: 10.1145/2783258.2788627

Asnat Greenstein-Messica, Lior Rokach, and Michael Friedman. 2017. Session-Based Recommendations Using Item Embedding. In Proceedings of the 22nd International Conference on Intelligent User Interfaces (IUI ‘17). Association for Computing Machinery, Limassol, Cyprus, 629–633. DOI: 10.1145/3025171.3025197

S. Hasanzadeh, S. M. Fakhrahmad, and M. Taheri. 2020. Review-Based Recommender Systems: A Proposed Rating Prediction Scheme Using Word Embedding Representation of Reviews. Comput. J. bxaa044, ; (2020), 1–10. DOI: 10.1093/comjnl/bxaa044

Antonio Hernando, JesÚs Bobadilla, and Fernando Ortega. 2016. A Non Negative Matrix Factorization for Collaborative Filtering Recommender Systems Based on a Bayesian Probabilistic Model. Knowledge-Based Systems 97, C (2016), 188—-202. DOI: 10.1016/j.knosys.2015.12.018

Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016. Session-Based Recommendations with Recurrent Neural Networks. In Proceedings of the International Conference on Learning Representations (ICLR 2016). OpenReview, San Juan, Puerto Rico, 1–10.

Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative Filtering for Implicit Feedback Datasets. In Proceedings of the 8th IEEE International Conference on Data Mining (ICDM ‘08). IEEE Computer Society, Pisa, Italy, 263–272. DOI: 10.1109/ICDM.2008.22

Salmo M.S. Júnior and Marcelo G. Manzato. 2015. Collaborative Filtering Based on Semantic Distance Among Items. In Proceedings of the 21st Brazilian Symposium on Multimedia and the Web (WebMedia ’15). Association for Computing Machinery, Manaus, Brazil, 53–56. DOI: 10.1145/2820426.2820466

Shan Khsuro, Zafar Ali, and Irfan Ullah. 2016. Recommender Systems: Issues, Challenges, and Research Opportunities. In Proceedings of the 7th International Conference on Information Science and Applications (ICISA 2016). Springer Science+Business Media, Ho Chi Minh, Vietnam, 1179–1189. DOI: 10.1007/978-981-10-0557-2_112

Jooeun Kim, Jinri Kim, Kwangeun Yeo, Eungi Kim, Kyoung-Woon On, Jonghwan Mun, and Joonseok Lee. 2024. General Item Representation Learning for Cold-start Content Recommendations. arXiv: 2404.13808 (2024), 1–14. DOI: 10.48550/arXiv.2404.13808

Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix Factorization Techniques For Recommender Systems. Computer 42, 8 (2009), 30–37. DOI: 10.1109/MC.2009.263

Quoc Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. In Proceedings of the 31st International Conference on Machine Learning (ICML 2014). JMLR.org, Beijing, China, 1188–1196. DOI: 10.5555/3044805.3045025

Pasquale Lisena, Albert Meroño-Peñuela, and Raphaëla Troncy. 2022. MIDI2vec: Learning MIDI embeddings for reliable prediction of symbolic music metadata. Semantic Web 13, 3 (2022), 357–377. DOI: 10.3233/SW-210446

Junling Liu, Chao Liu, Peilin Zhou, Qichen Ye, Dading Chong, Kang Zhou, Yueqi Xie, Yuwei Cao, Shoujin Wang, Chenyu You, and Philip S.Yu. 2023. LLM-Rec: Benchmarking Large Language Models on Recommendation Task. arXiv: 2308.12241 (2023), 1–13. DOI: 10.48550/arXiv.2308.12241

Jie Lu, Dianshuang Wu, Mingsong Mao, Wei Wang, and Guangquan Zhang. 2015. Recommender system application developments: A survey. Decision Support Systems 74 (2015), 12–32. DOI: 10.1016/j.dss.2015.03.008

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Conrado, and Jeffrey Dan. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS 2013). Curran Associates Inc., Stateline, NV, USA, 3111–3119. DOI: 10.5555/2999792.2999959

Cataldo Musto, Pasquale Lops, Marco de Gemmis, and Giovanni Semeraro. 2017. Semantics-aware Recommender Systems exploiting Linked Open Data and graph-based features. Knowledge-Based Systems 136 (2017), 1–14. DOI: 10.1016/j.knosys.2017.08.015

Makbule Gulcin Ozsoy. 2016. From Word Embeddings to Item Recommendation. arXiv: 1601.01356 (2016), 1–8. DOI: 10.48550/arXiv.1601.01356

Yuanyuan Qiu, Hongzheng Li, Shen Li, Yingdi Jiang, Renfen Hu, and Lijiao Yang. 2018. Revisiting Correlations between Intrinsic and Extrinsic Evaluations of Word Embeddings. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data (CCL 2018). Springer, Changsha, China, 209–221. DOI: 10.1007/978-3-030-01716-3_18

Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks (LREC 2010). European Language Resources Association (ELRA), Valletta, Malta, 45–50. DOI: 10.13140/2.1.2393.1847

Stefen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. In Proceedings WebMedia’2024, Juiz de Fora, Brazil Pedro R. Pires, Bruno B. Rizzi, and Tiago A. Almeida of the 25th Conference on Uncertainty in Artificial Intelligence (UAI ‘09). AUAI Press, Montreal, Canada, 452–461. DOI: 10.5555/1795114.1795167

Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. 2020. Neural Collaborative Filtering vs. Matrix Factorization Revisited. In Proceedings of the 14th ACM Conference on Recommender Systems (RecSys ’20). Association for Computing Machinery, Virtual Event, Brazil, 240–248. DOI: 10.1145/3383313.3412488

Steffen Rendle, Walid Krichene, Li Zhang, and Yehuda Koren. 2022. Revisiting the Performance of iALS on Item Recommendation Benchmarks. In Proceedings of the 16th ACM Conference on Recommender Systems (RecSys ’22). Association for Computing Machinery, Seattle, WA, USA, 427–435. DOI: 10.1145/3523227.3548486

Badrul M. Sarwar, George Karypis, Joseph A. Konstan, and John T. Riedl. 2000. Application of Dimensionality Reduction in Recommender System - A Case Study. In Proceedings of the 9th WebKDD Workshop on Web Mining for e-commerce (WebKDD ‘00). Association for Computing Machinery, Boston, Massachusetts, USA, 1–12. DOI: 10.21236/ada439541

Tobias Schnabel, Igor Labutov, David Mimno, and Thorsten Joachims. 2015. Evaluation methods for unsupervised word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015). Association for Computational Linguistics, Lisbon, Portugal, 298–307. DOI: 10.18653/v1/D15-1036

Guy Shani and Asela Gunawardana. 2011. Evaluating Recommendation Systems. In Recommender Systems Handbook, Francesco Ricci, Lior Rokach, Bracha Shapira, and Paul B. Kantor (Eds.). Springer US, New York, NY, USA, Chapter 8, 257–259. DOI: 10.1007/978-0-387-85820-3

Sumit Sidana, Mikhail Trofimov, Oleh Horodnytskyi, Charlotte Laclau, Yury Maximov, and Massih-Reza Amini. 2021. User preference and embedding learning with implicit feedback for recommender systems. Data Mining and Knowledge Discovery 35 (2021), 568–592. DOI: 10.1007/s10618-020-00730-8

Abe Vallerian Siswanto, Lilian Tjong, and Yordan Saputra. 2018. Simple Vector Representations of E-commerce Products. In 2018 International Conference on Asian Language Processing (IALP 2018). IEEE, Bandung, Indonesia, 368–372. DOI: 10.1109/IALP.2018.8629245

Yang Song, Lu Zhang, and Clyde Lee Giles. 2011. Automatic tag recommendation algorithms for social recommender systems. ACM Transactions on the Web 4, 1 (2011), 4:1–4:31. DOI: 10.1145/1921591.1921595

Jiaxi Tang and Ke Wang. 2018. Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining (WSDM ‘18). Association for Computing Machinery, Marina Del Rey, CA, USA, 565–573. DOI: 10.1145/2939672.2939673

Flavian Vasile, Elena Smirnova, and Alexis Conneau. 2016. Meta-Prod2Vec: Product Embeddings Using Side-Information for Recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys ‘16). Association for Computing Machinery, Boston, Massachusetts, USA, 225–232. DOI: 10.1145/2959100.2959160

Dongjing Wang, Guandong Xu, and Shuiguang Deng. 2017. Music recommendation via heterogeneous information graph embedding. In Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN 2017). IEEE, Anchorage, AK, USA, 596–603. DOI: 10.1109/IJCNN.2017.7965907

Jiaqi Wang and Jing Lv. 2020. Tag-informed collaborative topic modeling for cross domain recommendations. Knowledge-Based Systems 203 (2020), 106119. DOI: 10.1016/j.knosys.2020.106119

Qinyong Wang, Hongzhi Yin, Hao Wang, Quoc Viet Hung Nguyen, Zi Huang, and Lizhen Cui. 2019. Enhancing Collaborative Filtering with Generative Augmentation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’19). Association for Computing Machinery, Anchorage, AK, USA, 548–556. DOI: 10.1145/3292500.3330873

Tian Wang, Yuri M. Brovman, and Sriganesh Madhvanath. 2021. Personalized Embedding-based e-Commerce Recommendations at eBay. arXiv: 2102.06156 (2021), 1–9. DOI: 10.48550/arXiv.2102.06156

Heitor Werneck, Nícollas Silva, Matheus Carvalho Viana, Fernando Mourão, Adriano C. M. Pereira, and Leonardo Rocha. 2020. A Survey on Point-of-Interest Recommendation in Location-based Social Networks. In Proceedings of the Brazilian Symposium on Multimedia and the Web (WebMedia ’20). Association for Computing Machinery, São Luís, Brazil, 185–192. DOI: 10.1145/3428658.3430970

Dongqiang Yang, Ning Li, Li Zou, and Hongwei Ma. 2022. Lexical semantics enhanced neural word embeddings. Knowledge-Based Systems 252 (2022), 109298. DOI: 10.1016/j.knosys.2022.109298

Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Jundong Li, and Zi Huang. 2024. Self-Supervised Learning for Recommender Systems: A Survey. IEEE Transactions on Knowledge and Data Engineering 36 (2024), 335–355. DOI: 10.1109/TKDE.2023.3282907

Hafed Zarzour, Ziad A. Al-Sharif, and Yaser Jararweh. 2019. RecDNNing: a recommender system using deep neural network with user and item embeddings. In Proceedings of the 10th International Conference on Information and Communication Systems (ICICS 2019). IEEE, Irbid, Jordan, 99–103. DOI: 10.1109/IACS.2019.8809156

Fuzheng Zhang, Nicholas Jing Yuan, Defu Lian, Xing Xie, and Wei-Ying Ma. 2016. Collaborative Knowledge Base Embedding for Recommender Systems. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ‘16). Association for Computing Machinery, San Francisco, CA, USA, 353–362. DOI: 10.1145/2939672.2939673

Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep Learning Based Recommender System: A Survey and New Perspectives. ACM Comput. Surv. 52, 1 (2019), 5:1–5:35. DOI: 10.1145/3285029

Xiangyu Zhao, Maolin Wang, Xinjian Zhao, Jiansheng Li, Shucheng Zhou, Dawei Yin, Qing Li, Jiliang Tang, and Ruocheng Guo. 2023. Embedding in Recommender Systems: A Survey. arXiv: 2310.18608 (2023), 1–42. DOI: 10.48550/arXiv.2310.18608

Lütfi Kerem Şenel, İhsan Utlu, Veysel Yücesoy, Aykut Koç, and Tolga Çukur. 2018. Semantic Structure and Interpretability of Word Embeddings. IEEE/ACM Transactions on Audio, Speech, and Language Processing 26, 10 (2018), 1769–1779. DOI: 10.1109/TASLP.2018.2837384