Smart Notifications – An ML-based Framework to Boost User Engagement
Resumo
The use of daily push notifications is prevalent in many online and mobile applications to enhance and maintain user engagement. Push notifications are often used in customer relationship management (CRM) campaigns to promote engagement, and frequently a customer is subjected to several on a daily basis and many at the same time. This often results in multiple notifications being scheduled to a user simultaneously. Also, in online apps, push notifications can trigger new orders during various shifts throughout the day for each user. This paper presents a complete framework of push notification modeling that takes into account the human-in-the-loop aspect of the problem, mixing up modeling with business decisions. The model structure is based on a two-tower deep learning model to rank push notifications based on their relevance to users, utilizing push metadata and user features. It also analyzes the causal impact of sending push notifications during each shift of the day. We use it to successfully optimize more than 100 million daily push notifications on the food delivery app iFood, resulting in increased orders and reduced average push notifications per user.
Palavras-chave:
push notification, recommendation system, causal inference
Referências
Charu C Aggarwal 2016. Recommender Systems. Springer. https://doi.org/10.1007/978-3-319-29659-3
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901
Gautam Chauhan, Dhruva Vatsa Mishra, M Farida Begam, 2019. Customer-Aware Recommender System for Push Notifications in an e-commerce Environment. In 2019 Global Conference for Advancement in Technology (GCAT). IEEE, 1–7. https://doi.org/10.1109/GCAT47503.2019.8978330
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7–10. https://doi.org/10.1145/2988450.2988454
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191–198. https://doi.org/10.1145/2959100.2959190
Felipe Ferreira, Daniele R Souza, Igor Moura, Matheus Barbieri, and Hélio CV Lopes. 2020. Investigating multimodal features for video recommendations at globoplay. In Proceedings of the 14th ACM Conference on Recommender Systems. 571–572. https://doi.org/10.1145/3383313.3411553
Ronald Aylmer Fisher. 1936. Design of experiments. British Medical Journal 1, 3923 (1936), 554
Diana Gavilan and Gema Martinez-Navarro. 2022. Exploring user’s experience of push notifications: a grounded theory approach. Qualitative Market Research: An International Journal (2022). https://doi.org/10.1108/QMR-05-2021-0061
Merve Gençer, Gökhan Bilgin, Özgür Zan, and Tansel Voyvodaoğlu. 2013. A new framework for increasing user engagement in mobile applications using machine learning techniques. In Design, User Experience, and Usability. Web, Mobile, and Product Design: Second International Conference, DUXU 2013, Held as Part of HCI International 2013, Las Vegas, NV, USA, July 21-26, 2013, Proceedings, Part IV 2. Springer, 651–659. https://doi.org/10.1007/978-3-642-39253-5_72
Pierre Gutierrez and Jean-Yves Gérardy. 2017. Causal inference and uplift modelling: A review of the literature. In International conference on predictive applications and APIs. PMLR, 1–13
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web. 173–182. https://doi.org/10.1145/3038912.3052569
Fredrik Johansson, Uri Shalit, and David Sontag. 2016. Learning representations for counterfactual inference. In International conference on machine learning. PMLR, 3020–3029
Daniel Jurafsky and James H. Martin. 2009. Speech and Language Processing (2nd Edition). Prentice-Hall, Inc., USA
Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. 2017. Causal effect inference with deep latent-variable models. Advances in neural information processing systems 30 (2017)
Adithya Madhusoodanan, Anand Kumar, Kieran Fraser, and Bilal Yousuf. 2020. Machine Learning Approach to Manage Adaptive Push Notifications for Improving User Experience. In MobiQuitous 2020-17th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services. 488–493. https://doi.org/10.1145/3448891.3448956
Conor O’Brien, Huasen Wu, Shaodan Zhai, Dalin Guo, Wenzhe Shi, and Jonathan J Hunt. 2022. Should I send this notification? Optimizing push notifications decision making by modeling the future. arXiv preprint arXiv:2202.08812 (2022). https://doi.org/10.48550/arXiv.2202.08812
Tadashi Okoshi, Kota Tsubouchi, and Hideyuki Tokuda. 2019. Real-world product deployment of adaptive push notification scheduling on smartphones. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2792–2800. https://doi.org/10.1145/3292500.3330732
Kaare Brandt Petersen, Michael Syskind Pedersen, 2008. The matrix cookbook. Technical University of Denmark 7, 15 (2008), 510
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, 2018. Improving language understanding by generative pre-training. (2018)
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9
Mark J Schervish and Morris H DeGroot. 2012. Probability and statistics. Pearson Education
Uri Shalit, Fredrik D Johansson, and David Sontag. 2017. Estimating individual treatment effect: generalization bounds and algorithms. In International Conference on Machine Learning. PMLR, 3076–3085
Claudia Shi, David Blei, and Victor Veitch. 2019. Adapting Neural Networks for the Estimation of Treatment Effects. In Advances in Neural Information Processing Systems, H Wallach, H Larochelle, A Beygelzimer, F Alché-Buc, E Fox, and R Garnett (Eds.). Vol. 32. Curran Associates, Inc. [link].
Tian Wang, Yuri M Brovman, and Sriganesh Madhvanath. 2021. Personalized embedding-based e-commerce recommendations at ebay. arXiv preprint arXiv:2102.06156 (2021). https://doi.org/10.48550/arXiv.2102.06156
Kevin P Yancey and Burr Settles. 2020. A sleeping, recovering bandit algorithm for optimizing recurring notifications. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3008–3016. https://doi.org/10.1145/3394486.3403351
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901
Gautam Chauhan, Dhruva Vatsa Mishra, M Farida Begam, 2019. Customer-Aware Recommender System for Push Notifications in an e-commerce Environment. In 2019 Global Conference for Advancement in Technology (GCAT). IEEE, 1–7. https://doi.org/10.1109/GCAT47503.2019.8978330
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems. 7–10. https://doi.org/10.1145/2988450.2988454
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191–198. https://doi.org/10.1145/2959100.2959190
Felipe Ferreira, Daniele R Souza, Igor Moura, Matheus Barbieri, and Hélio CV Lopes. 2020. Investigating multimodal features for video recommendations at globoplay. In Proceedings of the 14th ACM Conference on Recommender Systems. 571–572. https://doi.org/10.1145/3383313.3411553
Ronald Aylmer Fisher. 1936. Design of experiments. British Medical Journal 1, 3923 (1936), 554
Diana Gavilan and Gema Martinez-Navarro. 2022. Exploring user’s experience of push notifications: a grounded theory approach. Qualitative Market Research: An International Journal (2022). https://doi.org/10.1108/QMR-05-2021-0061
Merve Gençer, Gökhan Bilgin, Özgür Zan, and Tansel Voyvodaoğlu. 2013. A new framework for increasing user engagement in mobile applications using machine learning techniques. In Design, User Experience, and Usability. Web, Mobile, and Product Design: Second International Conference, DUXU 2013, Held as Part of HCI International 2013, Las Vegas, NV, USA, July 21-26, 2013, Proceedings, Part IV 2. Springer, 651–659. https://doi.org/10.1007/978-3-642-39253-5_72
Pierre Gutierrez and Jean-Yves Gérardy. 2017. Causal inference and uplift modelling: A review of the literature. In International conference on predictive applications and APIs. PMLR, 1–13
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web. 173–182. https://doi.org/10.1145/3038912.3052569
Fredrik Johansson, Uri Shalit, and David Sontag. 2016. Learning representations for counterfactual inference. In International conference on machine learning. PMLR, 3020–3029
Daniel Jurafsky and James H. Martin. 2009. Speech and Language Processing (2nd Edition). Prentice-Hall, Inc., USA
Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. 2017. Causal effect inference with deep latent-variable models. Advances in neural information processing systems 30 (2017)
Adithya Madhusoodanan, Anand Kumar, Kieran Fraser, and Bilal Yousuf. 2020. Machine Learning Approach to Manage Adaptive Push Notifications for Improving User Experience. In MobiQuitous 2020-17th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services. 488–493. https://doi.org/10.1145/3448891.3448956
Conor O’Brien, Huasen Wu, Shaodan Zhai, Dalin Guo, Wenzhe Shi, and Jonathan J Hunt. 2022. Should I send this notification? Optimizing push notifications decision making by modeling the future. arXiv preprint arXiv:2202.08812 (2022). https://doi.org/10.48550/arXiv.2202.08812
Tadashi Okoshi, Kota Tsubouchi, and Hideyuki Tokuda. 2019. Real-world product deployment of adaptive push notification scheduling on smartphones. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2792–2800. https://doi.org/10.1145/3292500.3330732
Kaare Brandt Petersen, Michael Syskind Pedersen, 2008. The matrix cookbook. Technical University of Denmark 7, 15 (2008), 510
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, 2018. Improving language understanding by generative pre-training. (2018)
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9
Mark J Schervish and Morris H DeGroot. 2012. Probability and statistics. Pearson Education
Uri Shalit, Fredrik D Johansson, and David Sontag. 2017. Estimating individual treatment effect: generalization bounds and algorithms. In International Conference on Machine Learning. PMLR, 3076–3085
Claudia Shi, David Blei, and Victor Veitch. 2019. Adapting Neural Networks for the Estimation of Treatment Effects. In Advances in Neural Information Processing Systems, H Wallach, H Larochelle, A Beygelzimer, F Alché-Buc, E Fox, and R Garnett (Eds.). Vol. 32. Curran Associates, Inc. [link].
Tian Wang, Yuri M Brovman, and Sriganesh Madhvanath. 2021. Personalized embedding-based e-commerce recommendations at ebay. arXiv preprint arXiv:2102.06156 (2021). https://doi.org/10.48550/arXiv.2102.06156
Kevin P Yancey and Burr Settles. 2020. A sleeping, recovering bandit algorithm for optimizing recurring notifications. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3008–3016. https://doi.org/10.1145/3394486.3403351
Publicado
23/10/2023
Como Citar
PINTO, Victor M. Magalhães; OLIVEIRA, Leonardo M. Murtha; DE SOUSA, Ysabelle Pinheiro; DANYI, Stefania L.; LIMA, Dimas S..
Smart Notifications – An ML-based Framework to Boost User Engagement. In: SIMPÓSIO BRASILEIRO DE SISTEMAS MULTIMÍDIA E WEB (WEBMEDIA), 29. , 2023, Ribeirão Preto/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 25–31.
