Evaluating user experience in music discovery on deezer and spotify


A music recommendation system aims to proactively suggest items of interest to users based on their behavior, purpose, or previous listening. Traditionally, recommendations are evaluated by the performance of the algorithms, especially in prediction accuracy. However, researchers have recently started investigating system effectiveness using evaluation criteria aimed at the users' perspective. Research in the field of Recommendation Systems (RS) has demonstrated that considering user experience (UX) concepts in the recommendation generation process can impact in terms of user values and that besides the accuracy of the algorithms, other factors influence the music listening experience. These factors can involve system aspects, personal aspects, or situational aspects. In this sense, this work aims to present the results of an exploratory study conducted with two commercial music platforms (Spotify and Deezer). The study analyzed the interaction between music listeners with the two platforms. In addition, we checked how the user experience was related to other aspects (e.g., satisfaction, user activity, feedback, and others) that influenced listening preferences. The study was conducted with 10 participants. We used the Communicability Evaluation Method (CEM) to detect communicability failures focusing on the user and the Self-Assessment Manikin (SAM) questionnaire. The results show that the recommendation methods of the two platforms do not sufficiently consider the desirable aspects. Furthermore, users expressed dissatisfaction with the first recommendations received by the platforms.
Palavras-chave: Affective computing, Emotion recognition, Music information retrieval, Music recommendation, User experience


Willian G Assuncao, Lara SG Piccolo, and Luciana AM Zaina. 2022. Considering emotions and contextual factors in music recommendation: a systematic literature review. Multimedia Tools and Applications (2022), 1--41.

Rodrigo Bandeira-De-Mello. 2006. Softwares em pesquisa qualitativa. Pesquisa qualitativa em estudos organizacionais: paradigmas, estratégias e métodos. São Paulo: Saraiva 481 (2006), 241--266.

Dmitry Bogdanov, Martín Haro, Ferdinand Fuhrmann, Anna Xambó, Emilia Gómez, and Perfecto Herrera. 2013. Semantic audio content-based music recommendation and visualization based on user preference examples. Information Processing & Management 49, 1 (2013), 13--33.

Dirk Bollen, Bart P Knijnenburg, Martijn C Willemsen, and Mark Graus. 2010. Understanding choice overload in recommender systems. In Proceedings of the fourth ACM conference on Recommender systems. 63--70.

M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic diferential. J. Behav. Ther. & Exp. Psychiat. 25, I (1994), 49--59.

Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1 (1994), 49--59.

Jia-Wei Chang, Ching-Yi Chiou, Jia-Yi Liao, Ying-Kai Hung, Chien-Che Huang, Kuan-Cheng Lin, and Ying-Hung Pu. 2019. Music recommender using deep embedding-based features and behavior-based reinforcement learning. Multimedia Tools and Applications (2019), 1--28.

Chih-Ming Chen, Ming-Feng Tsai, Jen-Yu Liu, and Yi-Hsuan Yang. 2013. Music recommendation based on multiple contextual similarity information. In 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), Vol. 1. IEEE, 65--72.

Szu-Yu Chou, Yi-Hsuan Yang, and Yu-Ching Lin. 2015. Evaluating music recommendation in a real-world setting: On data splitting and evaluation metrics. In 2015 IEEE international conference on multimedia and expo (ICME). IEEE, 1--6.

Juliet M Corbin and Anselm Strauss. 1990. Grounded theory research: Procedures, canons, and evaluative criteria. Qualitative sociology 13, 1 (1990), 3--21.

Debashis Das, Laxman Sahoo, and Sujoy Datta. 2017. A survey on recommendation system. International Journal of Computer Applications 160, 7 (2017).

Willian Garcias de Assunção and Vania Paula de Almeida Neris. 2019. m-Motion: a mobile application for music recommendation that considers the desired emotion of the user. In Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems. 1--11.

J García de Quirós, Sandra Baldassarri, José Ramón Beltrán, A Guiu, and Pedro Álvarez. 2019. An Automatic Emotion Recognition System for Annotating Spotify's Songs. 345--362 pages.

Clarisse Sieckenius De Souza and Carla Faria Leitão. 2009. Semiotic engineering methods for scientific research in HCI. Synthesis Lectures on Human-Centered Informatics 2, 1 (2009), 1--122.

Deezer. 2021. ONDE A MÚSICA VIVE. https://www.deezer.com/br/company

Yashar Deldjoo, Markus Schedl, and Peter Knees. 2021. Content-based Music Recommendation: Evolution, State of the Art, and Challenges. arXiv preprint arXiv:2107.11803 (2021).

Bruce Ferwerda. 2016. Improving the user experience of music recommender systems through personality and cultural information. Ph.D. Dissertation. B. Ferwerda.

Bruce Ferwerda and Markus Schedl. 2014. Enhancing Music Recommender Systems with Personality Information and Emotional States: A Proposal.. In Umap workshops. 1--9.

Bruce Ferwerda, Markus Schedl, and Marko Tkalcic. 2015. Personality & Emotional States: Understanding Users' Music Listening Needs.. In UMAP Workshops.

Bruce Ferwerda, Andreu Vall, Marko Tkalcic, and Markus Schedl. 2016. Exploring music diversity needs across countries. In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization. 287--288.

Bruce Ferwerda, Emily Yang, Markus Schedl, and Marko Tkalcic. 2019. Personality and taxonomy preferences, and the influence of category choice on the user experience for music streaming services. Multimedia tools and applications 78, 14 (2019), 20157--20190.

Lavínia Matoso Freitas, Thiago Hellen O da Silva, and Marília Soares Mendes. 2016. Evaluation of spotify: an evaluation textual experience using the maltu methodology. In Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems. 1--4.

Jean Garcia-Gathright, Brian St. Thomas, Christine Hosey, Zahra Nazari, and Fernando Diaz. 2018. Understanding and evaluating user satisfaction with music discovery. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 55--64.

Jesse James Garrett. 2010. The elements of user experience: user-centered design for the web and beyond. Pearson Education.

G Geetha, M Safa, C Fancy, and D Saranya. 2018. A hybrid approach using collaborative filtering and content based filtering for recommender system. In Journal of Physics: Conference Series. IOP Publishing.

Yike Guo, Chao Wu, and Diego Peteiro-Barral. 2012. An EEG-Based brain informatics application for enhancing music experience. In International Conference on Brain Informatics. Springer, 265--276.

Byeong-Jun Han, Seungmin Rho, Sanghoon Jun, and Eenjun Hwang. 2010. Music emotion classification and context-based music recommendation. Multimedia Tools and Applications 47, 3 (2010), 433--460.

Joris H. Janssen, Van D. Broek, Egon L., and Joyce H.D.M. Westerink. 2012. Tune in to your emotions: A robust personalized affective music player. User Modeling and User-Adapted Interaction 22, 3 (2012), 255--279.

Elliot Jenkins and Yanyan Yang. 2016. Creating a music recommendation and streaming application for android. In International Conference on Database and Expert Systems Applications. Springer, 201--215.

Chengkun Jiang and Yuan He. 2016. Smart-dj: Context-aware personalization for music recommendation on smartphones. In 2016 IEEE 22nd International Conference on Parallel and Distributed Systems (ICPADS). IEEE, 133--140.

Yucheng Jin, Nyi Nyi Htun, Nava Tintarev, and Katrien Verbert. 2019. ContextPlay: Evaluating User Control for Context-Aware Music Recommendation. In Proceedings of the 27th ACM Conference on User Modeling, Adaptation and Personalization. 294--302.

Yucheng Jin, Nava Tintarev, Nyi Nyi Htun, and Katrien Verbert. 2020. Effects of personal characteristics in control-oriented user interfaces for music recommender systems. User Modeling and User-Adapted Interaction 30, 2 (2020), 199--249.

Oliver P John, Eileen M Donahue, and Robert L Kentle. 1991. The big five inventory---versions 4a and 54.

Youngmoo E Kim, Erik M Schmidt, Raymond Migneco, Brandon G Morton, Patrick Richardson, Jeffrey Scott, Jacquelin A Speck, and Douglas Turnbull. 2010. Music emotion recognition: A state of the art review. In Proc. ISMIR. 255--266.

Bart P Knijnenburg, Martijn C Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the user experience of recommender systems. User Modeling and User-Adapted Interaction 22, 4 (2012), 441--504.

Joseph A Konstan and John Riedl. 2012. Recommender systems: from algorithms to user experience. User modeling and user-adapted interaction 22, 1--2 (2012), 101--123.

Bozena Kostek. 2018. Listening to live music: life beyond music recommendation systems. In 2018 Joint Conference-Acoustics. IEEE, 1--5.

Effie Lai-Chong Law, Virpi Roto, Marc Hassenzahl, Arnold POS Vermeeren, and Joke Kort. 2009. Understanding, scoping and defining user experience: a survey approach. In Proceedings of the SIGCHI conference on human factors in computing systems. 719--728.

Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. Research methods in human-computer interaction. Morgan Kaufmann.

Jin Ha Lee and Rachel Price. 2016. User experience with commercial music services: An empirical exploration. Journal of the Association for Information Science and Technology 67, 4 (2016), 800--811.

Jin Ha Lee, Rachel Wishkoski, Lara Aase, Perry Meas, and Chris Hubbles. 2017. Understanding users of cloud music services: selection factors, management and access behavior, and perceptions. Journal of the Association for Information Science and Technology 68, 5 (2017), 1186--1200.

Wei-Po Lee, Chun-Ting Chen, Jhih-Yuan Huang, and Jhen-Yi Liang. 2017. A smartphone-based activity-aware system for music streaming recommendation. Knowledge-Based Systems 131 (2017), 70--82.

Qiuxia Li and Dan Liu. 2017. Research of music recommendation system based on user behavior analysis and word2vec user emotion extraction. In International Conference on Intelligent and Interactive Systems and Applications. Springer, 469--475.

Maake Benard Magara, Sunday Ojo, Seleman Ngwira, and Tranos Zuva. 2016. MPlist: Context aware music playlist. In 2016 IEEE International Conference on Emerging Technologies and Innovative Business Practices for the Transformation of Societies (EmergiTech). IEEE, 309--316.

Daniel Müllensiefen, Bruno Gingras, Jason Musil, and Lauren Stewart. 2014. The musicality of non-musicians: an index for assessing musical sophistication in the general population. PloS one 9, 2 (2014), e89642.

Karla Okada, Börje F. Karlsson, Laura Sardinha, and Tomaz Noleto. 2013. ContextPlayer: Learning contextual music preferences for situational recommendations. SIGGRAPH Asia 2013 Symposium on Mobile Graphics and Interactive Applications on - SA '13 (2013), 1--7.

Martin Pichl, Eva Zangerle, and Günther Specht. 2014. Combining Spotify and Twitter Data for Generating a Recent and Public Dataset for Music Recommendation.. In Grundlagen von Datenbanken. 35--40.

Johnny Saldaña. 2021. The coding manual for qualitative researchers. sage.

Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web. 285--295.

Markus Schedl. 2013. Ameliorating music recommendation: Integrating music content, music context, and user context for improved music retrieval and recommendation. In Proceedings of International Conference on Advances in Mobile Computing & Multimedia. 3--9.

Elena Shakirova. 2017. Collaborative filtering for music recommender system. In 2017 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus). IEEE, 548--550.

Yading Song, Simon Dixon, and Marcus Pearce. 2012. A survey of music recommendation systems and future perspectives. In 9th International Symposium on Computer Music Modeling and Retrieval, Vol. 4. Citeseer, 395--410.

Spotify. 2021. Company Info. https://newsroom.spotify.com/company-info/

B Srikanth and V Nagalakshmi. 2020. Songs Recommender System using Machine Learning Algorithm: SVD Algorithm. Int. J. Innov. Sci. & Res. Tech 5 (2020), 390--392.

Muh-Chyun Tang and Mang-Yuan Yang. 2017. Evaluating Music Discovery Tools on Spotify: The Role of User Preference Characteristics. Journal of Library & Information Studies 15, 1 (2017).

Yuan-Ching Teng, Ying-Shu Kuo, and Yi-Hsuan Yang. 2013. A large in-situ dataset for context-aware music recommendation on smartphones. In 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW). IEEE, 1--4.

Julián Urbano, Markus Schedl, and Xavier Serra. 2013. Evaluation in music information retrieval. Journal of Intelligent Information Systems 41, 3 (2013), 345--369.

Maarten W van Someren, Yvonne F Barnard, and Jacobijn A C Sandberg. 1994. The Think Aloud Method: A practical guide to modelling cognitive processes. Academic Press, London, UK. 218 pages.

Arnold POS Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout, and Kaisa Väänänen-Vainio-Mattila. 2010. User experience evaluation methods: current state and development needs. In Proceedings of the 6th Nordic conference on human-computer interaction: Extending boundaries. 521--530.

Chen-Ya Wang, Yu-Chi Wang, and Seng-Cho T Chou. 2018. A context and emotion aware system for personalized music recommendation. Journal of Internet Technology 19, 3 (2018), 765--779.

Xinxi Wang, David Rosenblum, and Ye Wang. 2012. Context-aware mobile music recommendation for daily activities. In Proceedings of the 20th ACM international conference on Multimedia. 99--108.

Xinxi Wang and Ye Wang. 2014. Improving content-based and hybrid music recommendation using deep learning. In Proceedings of the 22nd ACM international conference on Multimedia. 627--636.

Jéssica Adriane Weigsding and Carmem Patrícia Barbosa. 2014. A influência da música no comportamento humano. Arquivos do MUDI, Maringá 18, 2 (2014), 47--62.

Martijn C Willemsen, Bart P Knijnenburg, Mark P Graus, LC Velter-Bremmers, and Kai Fu. 2011. Using latent features diversification to reduce choice difficulty in recommendation lists. RecSys 11, 2011 (2011), 14--20.
ASSUNÇÃO, Willian Garcias de; ZAINA, Luciana Aparecida Martinez. Evaluating user experience in music discovery on deezer and spotify. In: SIMPÓSIO BRASILEIRO SOBRE FATORES HUMANOS EM SISTEMAS COMPUTACIONAIS (IHC), 21. , 2022, Diamantina. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2022 .