Detectando Incoerências Avaliativas em E-commerce com LLMs - Um Estudo de Caso na Amazon Brasil - Marreira, Emanuelle; Melo, Tiago de; Oliveira, Miguel; Maurício, Carlos (Proceedings of the Brazilian Symposium on Multimedia and the Web (WebMedia); 2025: Proceedings of the 31st Brazilian Symposium on Multimedia and the Web, 2025)
This study evaluates the ability of large language models (LLMs) to detect incoherence between the text of product reviews and their assigned rating (1 or 5 stars). Using ChatGPT-o3 in five independent runs, we observed high variability in labeling and low overall agreement (Fleiss’ 𝜅 = 0.177). A conservative approach selected 415 reviews unanimously labeled as incoherent, which were subsequently submitted for human evaluation. The agreement between human annotators was substantial (Cohen’s 𝜅 = 0.709), allowing the isolation of 231 cases with a clear sentiment judgment. The comparison showed that only 28.1% of the LLM classifications matched the human judgment. These results suggest that, while promising, LLMs still require rigorous validation and careful calibration for critical semantic interpretation tasks.