Comparative Analysis of Collaborative Filtering-Based Predictors of Scores in Surveys of a Large Company

• Markos F. B. G. Oliveira UTFPR / Pin People Tecnologia Aplicada a Pessoas
• Myriam Delgado UTFPR
• Ricardo Lüders UTFPR

DOI: https://doi.org/10.5753/eniac.2021.18299

Resumo

Collaborative Filtering (CF) can be understood as the process of predicting the preferences of users and deriving useful patterns by studying their activities. In the survey context, it can be used to predict answers to questions as combinations of other available answers. In this paper, we aim to test five CF-based algorithms (item-item, iterative matrix factorization, neural collaborative filtering, logistic matrix factorization, and an ensemble of them) to estimate scores in four survey applications (checkpoints) composed of 700,000 employee's ratings. These data have been collected from 2019 to 2020 by a large Brazilian tech company with more than 10,000 employees. The results show that collaborative filtering approaches provide relevant alternatives to score questions of surveys. They provided good quality estimates. This result can be further explored to eventually reduce the size of questionnaires, avoiding burden phenomena faced by respondents when dealing with large surveys.

Referências

Al-Shamri, M. Y. H. (2016). User profiling approaches for demographic recommender systems. Knowledge-Based Systems, 100:175–187.

Boim, R., Greenshpan, O., Milo, T., Novgorodov, S., Polyzotis, N., and Tan, W.-C. In 2012 IEEE 28th In(2012). Asking the right questions in crowd data sourcing. ternational Conference on Data Engineering, pages 1261–1264. IEEE.

Bokde, D., Girase, S., and Mukhopadhyay, D. (2015). Matrix factorization model in collaborative filtering algorithms: A survey. Procedia Computer Science, 49:136–146.

Chun, A. Y., Heeringa, S., and Schouten, J. (2018). Responsive and adaptive design for survey optimization. Journal of Official Statistics, 34(3):581–597.

Early, K., Mankoff, J., and Fienberg, S. E. (2017). Dynamic question ordering in online surveys. Journal of Official Statistics, 33.

Gonzalez, J. M. and Eltinge, J. L. (2008). Adaptive matrix sampling for the consumer expenditure quarterly interview survey. In Proceedings of the Section on Survey Research Methods, American Statistical Association, pages 2081–8.

He, H. and Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9):1263–1284.

He, X., Liao, L., Zhang, H., Nie, L., Hu, X., and Chua, T. S. (2017). Neural collaborative filtering. pages 173–182. International World Wide Web Conf. Steering Committee.

Huang, J. and Ling, C. X. (2005). Using auc and accuracy in evaluating learning algorithms. IEEE Transactions on knowledge and Data Engineering, 17(3):299–310.

Jahrer, M., Töscher, A., and Legenstein, R. (2010). Combining predictions for accurate recommender systems. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 693–702.

Johnson, C. C. (2014). Logistic matrix factorization for implicit feedback data. Advances in Neural Information Processing Systems, 27(78):1–9.

Koren, Y. (2009). Collaborative filtering with temporal dynamics. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 447–456.

Krosnick, J. A. (2018). Questionnaire design. In The Palgrave handbook of survey research, pages 439–455. Springer.

Kulkarni, P. V., Rai, S., and Kale, R. (2020). Recommender system in elearning: a survey. In Proceeding of International Conference on Computational Science and Applications, pages 119–126. Springer.

Lavrakas, P. J. (2008). Encyclopedia of survey research methods. Sage publications.

Li, S., Karatzoglou, A., and Gentile, C. (2016). Collaborative filtering bandits. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 539–548.

Lika, B., Kolomvatsos, K., and Hadjiefthymiades, S. (2014). Facing the cold start problem in recommender systems. Expert Systems with Applications, 41(4):2065–2073.

Melville, P., Mooney, R. J., Nagarajan, R., et al. (2002). Content-boosted collaborative filtering for improved recommendations. Aaai/iaai, 23:187–192.

Ortigosa, A., Paredes, P., and Rodriguez, P. (2010). Ah-questionnaire: An adaptive hierarchical questionnaire for learning styles. Computers & Education, 54(4):999–1005.

Pazzani, M. J. and Billsus, D. (2007). Content-based recommendation systems. In The adaptive web, pages 325–341. Springer.

Rendle, S., Krichene, W., Zhang, L., and Anderson, J. (2020). Neural collaborative filtering vs. matrix factorization revisited.

Salakhutdinov, R., Mnih, A., and Hinton, G. (2007). Restricted boltzmann machines for collaborative filtering. In Proceedings of the 24th international conference on Machine learning, pages 791–798.

Schouten, B., Calinescu, M., and Luiten, A. (2013). Optimizing quality of response through adaptive survey designs. Survey Methodology, 39(1):29–58.

Song, L., Tekin, C., and Van Der Schaar, M. (2014). Online learning in large-scale contextual recommender systems. IEEE Transactions on Services Computing, 9(3):433–445.

Su, X. and Khoshgoftaar, T. M. (2009). A survey of collaborative filtering techniques. Advances in artificial intelligence, 2009.

Wagner, J. R. (2008). Adaptive survey design to reduce nonresponse bias. PhD thesis, University of Michigan.

Wang, H., Wu, Q., and Wang, H. (2017). Factorization bandits for interactive recommendation. In Thirty-First AAAI Conference on Artificial Intelligence.

Wu, C., Wu, F., An, M., Huang, J., Huang, Y., and Xie, X. (2019). Npa: Neural news recommendation with personalized attention. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2576–2584.

Wu, Q., Wang, H., Gu, Q., and Wang, H. (2016). Contextual bandits in a collaborative In Proceedings of the 39th International ACM SIGIR conference on environment. Research and Development in Information Retrieval, pages 529–538.

Zhang, C., Taylor, S. J., Cobb, C., and Sekhon, J. (2020). Active matrix factorization for surveys. Annals of Applied Statistics, 14.