Automated Topic Annotation in Brazilian Product Reviews: A Case Study of Adversarial Examples with Sabia-3
Resumo
High-quality annotated data is essential for many Natural Language Processing tasks, but traditional human annotation methods are often resource-intensive. Large Language Models (LLMs) offer potential solutions by generating labels for training datasets. This paper explores the effectiveness of using the Sabiá-3 LLM for automatically labeling data for a multi-label topic classification task in Brazilian Portuguese product reviews. We compare the performance of Sabia-3-generated labels against human annotations using the RePro dataset. The study evaluates Sabiá-3 on both random and adversarial datasets, highlighting its strengths in frequent topics, while identifying limitations in more nuanced categories. Models trained on Sabiá-3 annotations showed promising results in common categories but faced challenges with ambiguous cases. Our findings suggest that while LLMs can streamline parts of the annotation process, human oversight remains essential, particularly in complex or less frequent cases. This research contributes new insights into the use of LLMs for automated data annotation in Brazilian Portuguese.
Referências
Ding, B., Qin, C., Liu, L., Chia, Y. K., Li, B., Joty, S., and Bing, L. (2023). Is gpt-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173–11195.
dos Santos Silva, L. N., Real, L., Zandavalle, A. C. B., Rodrigues, C. F. G., da Silva Gama, T., Souza, F. G., and Zaidan, P. D. S. (2024). RePro: a benchmark for opinion mining for Brazilian Portuguese. In Gamallo, P., Claro, D., Teixeira, A., Real, L., Garcia, M., Oliveira, H. G., and Amaro, R., editors, Proceedings of the 16th International Conference on Computational Processing of Portuguese- Vol. 1, pages 432–440, Santiago de Compostela, Galicia/Spain. Association for Computational Lingustics.
Fonseca, E., Oliveira, A., Gadelha, C., and Guandaline, V. (2020). Brands.br - a portuguese reviews corpus. In OpenCor.
Inacio, M. L. and Oliveira, H. G. (2024). Generation of punning riddles in portuguese with prompt chaining paper type: Late breaking results. 15th International Conference on Computational Creativity (ICCC’24).
Pollach, I. (2006). Electronic word of mouth: A genre analysis of product reviews on consumer opinion web sites. In Proceedings of the 39th Annual Hawaii International Conference on System Sciences (HICSS’06), volume 3, pages 51c–51c.
Real, L., Bento, A., Soares, K., Oshiro, M., and Mafra, A. (2020). B2w-reviews02, an annotated review sample. In OpenCor, 2020.
Real, L., Oshiro, M., and Mafra, A. (2019). B2w-reviews01-an open product reviews corpus. In the Proceedings of the XII Symposium in Information and Human Language Technology, pages 200–208.
Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, T., and Kong, L. (2022). Zerogen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11653–11669.
Zagatti, F., Silva, L., and Real, L. (2021). Anonymization of the b2w-reviews01 corpus. In OpenCor.
