Enhanced Analysis of User Perceptions through Natural Language Processing Approaches

• Ana Cláudia Machado UFSJ
• Gabriel Prenassi UFSJ
• Elisa Tuler UFSJ / MGI
• Leonardo Rocha UFSJ

DOI: https://doi.org/10.5753/webmedia_estendido.2025.13901

Abstract

In this article, we present a framework for analyzing users’ textual comments, going beyond the evaluation carried out using the System Usability Scale, a traditional methodology for system assessment. The framework performs textual analysis through Topic Modeling, Text Summarization, and Sentiment Analysis. Topic Modeling identifies semantic topics in the comments, while Text Summarization generates summaries of each topic, enabling explainability. Sentiment Analysis, in turn, categorizes the sentiment of each topic, identifying its polarity. We propose an intuitive visual interface and apply our framework in a real-world scenario, where it enabled more accurate feedback.

Keywords: Modelagem de Tópicos, Análise de Sentimento, LLMs, SUS

References

Claudio Moisés Valiense De Andrade, Washington Cunha, Guilherme Fonseca, Ana Clara Pagano, Luana De Castro Santos, Adriana Silvina Pagano, Leonardo Rocha, and Marcos André Gonçalves. 2024. Explaining the Hardest Errors of Contextual Embedding Based Classifiers. In Proceedings of the 28th ConLL.

John Brooke. 1996. A quick and dirty usability scale. Usability Evaluation in Industry (1996), 189–194.

James Eschrich and Sarah Sterman. 2024. A Framework For Discussing LLMs as Tools for Qualitative Analysis. ArXiv abs/2407.11198 (2024).

Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, and Simon Tangi Perrault. 2024. CollabCoder: A Lower-barrier, Rigorous Workflow for Inductive Collaborative Qualitative Analysis with Large Language Models. In Proceedings of the 2024 CHI. DOI: 10.1145/3613904.3642002

Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794 (2022).

Kai-Robin Lange, Jonas Rieger, and Carsten Jentsch. 2022. Lex2Sent: A bagging approach to unsupervised sentiment analysis. arXiv:2209.13023 (2022).

Washington Luiz, Felipe Viegas, Rafael Alencar, Fernando Mourão, Thiago Salles, Dárlinton Carvalho, Marcos Andre Gonçalves, and Leonardo Rocha. 2018. A feature-oriented sentiment rating for mobile app reviews. In Proc. of the WWW.

Ali Nouri, Beatriz Cabrero-Daniel, Fredrik Torner, Hakan Sivencrona, and Christian Berger. 2024. Welcome Your New AI Teammate: On Safety Analysis by Leashing Large Language Models. In Proceedings of the IEEE/ACM CAIN ’24.

Antônio Pereira, Pablo Cecilio, Felipe Viegas, Washington Cunha, Elisa Tuler, and Leonardo Rocha. 2022. Evaluating topic modeling pre-processing pipelines for portuguese texts. In Proceedings of the WebMedia. 191–201.

Chau Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, and Mohit Iyyer. 2024. TopicGPT: A Prompt-based Topic Modeling Framework. In Proceedings of the 2024 Conference of the North American Chapter of the ACL. 2956–2984.

Felipe Viegas, Sérgio Canuto, Christian Gomes, Washington Luiz, Thierson Rosa, Sabir Ribas, Leonardo Rocha, and Marcos André Gonçalves. 2019. CluWords: exploiting semantic word clustering representation for enhanced topic modeling. In Proceedings of the 12a ACM WSDM. 753–761.

Tianhao Zhang, Fu Peiguo, Jie Liu, Yihe Zhang, and Xingmei Chen. 2024. NLDesign: A UI Design Tool for Natural Language Interfaces. In Proc.of the ACM-TURC.