A Pipeline for Monitoring and Maintaining a Text Classification Tool in Production

• Elene F. Ohata UFC
• César Lincoln C. Mattos UFC
• Paulo Antonio L. Rêgo UFC

DOI: https://doi.org/10.5753/semish.2024.2438

Resumo

Text classification has been a core component of several applications. Modern machine learning operations strategies address challenges in deploying and maintaining models in production environments. In this work, we describe and experiment with a pipeline for monitoring and updating a text classification tool deployed in a major information technology company. The proposed fully automatic approach also enables visual inspection of its operations via dashboards. The solution is thoroughly evaluated in two experimental scenarios: a static one, focusing on the Natural Language Processing (NLP) and Machine Learning (ML) stages to build the text classifier; and a dynamic one, where the pipeline enables automatic model updates. The obtained results are promising and indicate the validity of the implemented methodology.

Referências

Alla, S., Adari, S. K., Alla, S., and Adari, S. K. (2021). What is mlops? Beginning MLOps with MLFlow: Deploy Models in AWS SageMaker, Google Cloud, and Microsoft Azure, pages 79–124.

Arias-Barahona, M. X., Arteaga-Arteaga, H. B., Orozco-Arias, S., Flórez-Ruíz, J. C., Valencia-Díaz, M. A., and Tabares-Soto, R. (2023). Requests classification in the customer service area for software companies using machine learning and natural language processing. PeerJ Computer Science, 9:e1016.

Borg, A., Boldt, M., Rosander, O., and Ahlstrand, J. (2021). E-mail classification with machine learning and word embeddings for improved customer support. Neural Computing and Applications, 33(6):1881–1902.

Cahyani, D. E. and Patasik, I. (2021). Performance comparison of tf-idf and word2vec models for emotion text classification. Bulletin of Electrical Engineering and Informatics, 10(5):2780–2788.

Cawley, G. C. and Talbot, N. L. (2010). On over-fitting in model selection and subsequent selection bias in performance evaluation. The Journal of Machine Learning Research, 11:2079–2107.

Essa, E., Omar, K., and Alqahtani, A. (2023). Fake news detection based on a hybrid bert and lightgbm models. Complex & Intelligent Systems, pages 1–12.

Gift, N. and Deza, A. (2021). Practical MLOps. O’Reilly Media, Inc.

Haq, M. A., Khan, M. A. R., and Alshehri, M. (2022). Insider threat detection based on nlp word embedding and machine learning. Intell. Autom. Soft Comput, 33:619–635.

Kaminwar, S. R., Goschenhofer, J., Thomas, J., Thon, I., and Bischl, B. (2021). Structured verification of machine learning models in industrial settings. Big Data.

Kim, Y. (2014). Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).

Mäkinen, S., Skogström, H., Laaksonen, E., and Mikkonen, T. (2021). Who needs mlops: What data scientists seek to accomplish and how can mlops help? In 2021 IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN), pages 109–112. IEEE.

Nigenda, D., Karnin, Z., Zafar, M. B., Ramesha, R., Tan, A., Donini, M., and Kenthapadi, K. (2022). Amazon sagemaker model monitor: A system for real-time insights into deployed machine learning models. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3671–3681.

Paleyes, A., Urma, R.-G., and Lawrence, N. D. (2022). Challenges in deploying machine learning: a survey of case studies. ACM Computing Surveys, 55(6):1–29.

Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., and Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in neural information processing systems, 28.

Studer, S., Bui, T. B., Drescher, C., Hanuschkin, A., Winkler, L., Peters, S., and Müller, K.-R. (2021). Towards crisp-ml (q): a machine learning process model with quality assurance methodology. Machine learning and knowledge extraction, 3(2):392–413.

Symeonidis, G., Nerantzis, E., Kazakis, A., and Papakostas, G. A. (2022). Mlopsdefinitions, tools and challenges. In 2022 IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC), pages 0453–0460. IEEE.

van de Ven, G. M., Tuytelaars, T., and Tolias, A. S. (2022). Three types of incremental learning. Nature Machine Intelligence, 4(12):1185–1197.

Wainer, J. and Cawley, G. (2021). Nested cross-validation when selecting classifiers is overzealous for most practical applications. Expert Systems with Applications, 182:115222.