Contextual stance classification using prompt engineering
Resumo
This paper introduces a prompt-based method for few-shot learning addressing, as an application example, contextual stance classification, that is, the task of determining the attitude expressed by a given statement within a conversation thread with multiple points of view towards another statement. More specifically, we envisaged a method that uses the existing conversation thread (i.e., messages that are part of the test data) to create natural language prompts for few-shot learning with minimal reliance on training samples, whose preliminary results suggest that prompt engineering may be a competitive alternative to supervised methods both in terms of accuracy and development costs for the task at hand.
Palavras-chave:
stance classification, ChatGPT, prompt engineering
Referências
ALDayel, A. and Magdy, W. (2021). Stance detection on social media: State of the art and trends. Information Processing & Management, 58(4):102597.
Alhindi, T., Alabdulkarim, A., Alshehri, A., Abdul-Mageed, M., and Nakov, P. (2021). AraStance: A multi-country and multi-domain dataset of Arabic stance detection for fact checking. In 4th Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 57–65, Online. Assoc. for Computational Linguistics.
Bahuleyan, H. and Vechtomova, O. (2017). UWaterloo at SemEval-2017 task 8: Detecting stance towards rumours with topic independent features. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 461–464, Vancouver, Canada. Association for Computational Linguistics.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners.
Chen, N., Chen, X., and Pang, J. (2022). A multilingual dataset of covid-19 vaccination attitudes on twitter. Data in Brief, 44:108503.
Derczynski, L., Bontcheva, K., Liakata, M., Procter, R., Wong Sak Hoi, G., and Zubi aga, A. (2017). SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 69–76, Vancouver, Canada. Association for Computational Linguistics.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.
Fajcik, M., Smrz, P., and Burget, L. (2019). BUT-FIT at SemEval-2019 task 7: Determining the rumour stance with pre-trained deep bidirectional transformers. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1097–1104, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Gohring, A., Klenner, M., and Conrad, S. (2021). DeInStance: Creating and evaluating a german corpus for fine-grained inferred stance detection. In 17th Conference on Nat ural Language Processing (KONVENS 2021), pages 213–217, Düsseldorf, Germany. KONVENS 2021 Organizers.
Gorrell, G., Bontcheva, K., Derczynski, L., Kochkina, E., Liakata, M., and Zubiaga, A. (2018). Rumoureval 2019: Determining rumour veracity and support for rumours.
Jaziriyan, M. M., Akbari, A., and Karbasi, H. (2021). ExaASC: A General Target Based Stance Detection Corpus in Arabic Language. In 11th International Conference on Computer Engineering and Knowledge (ICCKE), pages 424–429, Mashhad, Iran. IEEE.
Kochkina, E., Liakata, M., and Augenstein, I. (2017). Turing at SemEval-2017 task 8: Sequential approach to rumour stance classification with branch-LSTM. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 475–480, Vancouver, Canada. Association for Computational Linguistics.
Li, Q., Zhang, Q., and Si, L. (2019). eventAI at SemEval-2019 task 7: Rumor detection on social media by exploiting content, user credibility and propagation information. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 855–859, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Li, Y., Sosea, T., Sawant, A., Nair, A. J., Inkpen, D., and Caragea, C. (2021). P-stance: A large dataset for stance detection in political domain. In Findings of ACL-IJCNLP 2021, pages 2355–2365, Online. Assoc. for Computational Linguistics.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9).
Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X., and Cherry, C. (2016). SemEval 2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41, San Diego, California. Association for Computational Linguistics.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners.
Sakketou, F., Lahnala, A., Vogel, L., and Flek, L. (2022). Investigating user radicalization: A novel dataset for identifying fine-grained temporal shifts in opinion. In LREC-2022 proceedings, pages 3798–3808, Marseille, France. ELRA.
Tutek, M., Sekulić, I., Gombar, P., Paljak, I., Čulinović, F., Boltužić, F., Karan, M., Alagić, D., and Šnajder, J. (2016). TakeLab at SemEval-2016 task 6: Stance classification in tweets using a genetic algorithm based ensemble. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 464–468, San Diego, California. Association for Computational Linguistics.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, pages 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
Wang, F., Lan, M., and Wu, Y. (2017). ECNU at SemEval-2017 task 8: Rumour evaluation using effective features and supervised ensemble models. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 491–496, Vancouver, Canada. Association for Computational Linguistics.
Wei, W., Zhang, X., Liu, X., Chen, W., and Wang, T. (2016). pkudblab at SemEval-2016 task 6 : A specific convolutional neural network system for effective stance detection. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), pages 384–388, San Diego, California. Association for Computational Linguistics.
Won, M. and Fernandes, J. (2022). SS-PT: A stance and sentiment data set from Portuguese quoted tweets. In PROPOR-2022 proceedings, pages 110–121, Fortaleza, Brazil. Springer.
Yang, R., Xie, W., Liu, C., and Yu, D. (2019). BLCU NLP at SemEval-2019 task 7: An inference chain-based GPT model for rumour evaluation. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1090–1096, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Yin, W., Hay, J., and Roth, D. (2019). Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach.
Zarrella, G. and Marsh, A. (2016). MITRE at SemEval-2016 task 6: Transfer learning for stance detection. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 458–463, San Diego, California. Association for Computational Linguistics.
Zhang, B., Ding, D., and Jing, L. (2023). How would Stance Detection Techniques Evolve after the Launch of ChatGPT?
Zhang, L., Wang, S., and Liu, B. (2018). Deep learning for sentiment analysis: A survey. WIREs Data Mining and Knowledge Discovery, 8(4):e1253.
Alhindi, T., Alabdulkarim, A., Alshehri, A., Abdul-Mageed, M., and Nakov, P. (2021). AraStance: A multi-country and multi-domain dataset of Arabic stance detection for fact checking. In 4th Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 57–65, Online. Assoc. for Computational Linguistics.
Bahuleyan, H. and Vechtomova, O. (2017). UWaterloo at SemEval-2017 task 8: Detecting stance towards rumours with topic independent features. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 461–464, Vancouver, Canada. Association for Computational Linguistics.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners.
Chen, N., Chen, X., and Pang, J. (2022). A multilingual dataset of covid-19 vaccination attitudes on twitter. Data in Brief, 44:108503.
Derczynski, L., Bontcheva, K., Liakata, M., Procter, R., Wong Sak Hoi, G., and Zubi aga, A. (2017). SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 69–76, Vancouver, Canada. Association for Computational Linguistics.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.
Fajcik, M., Smrz, P., and Burget, L. (2019). BUT-FIT at SemEval-2019 task 7: Determining the rumour stance with pre-trained deep bidirectional transformers. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1097–1104, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Gohring, A., Klenner, M., and Conrad, S. (2021). DeInStance: Creating and evaluating a german corpus for fine-grained inferred stance detection. In 17th Conference on Nat ural Language Processing (KONVENS 2021), pages 213–217, Düsseldorf, Germany. KONVENS 2021 Organizers.
Gorrell, G., Bontcheva, K., Derczynski, L., Kochkina, E., Liakata, M., and Zubiaga, A. (2018). Rumoureval 2019: Determining rumour veracity and support for rumours.
Jaziriyan, M. M., Akbari, A., and Karbasi, H. (2021). ExaASC: A General Target Based Stance Detection Corpus in Arabic Language. In 11th International Conference on Computer Engineering and Knowledge (ICCKE), pages 424–429, Mashhad, Iran. IEEE.
Kochkina, E., Liakata, M., and Augenstein, I. (2017). Turing at SemEval-2017 task 8: Sequential approach to rumour stance classification with branch-LSTM. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 475–480, Vancouver, Canada. Association for Computational Linguistics.
Li, Q., Zhang, Q., and Si, L. (2019). eventAI at SemEval-2019 task 7: Rumor detection on social media by exploiting content, user credibility and propagation information. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 855–859, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Li, Y., Sosea, T., Sawant, A., Nair, A. J., Inkpen, D., and Caragea, C. (2021). P-stance: A large dataset for stance detection in political domain. In Findings of ACL-IJCNLP 2021, pages 2355–2365, Online. Assoc. for Computational Linguistics.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9).
Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X., and Cherry, C. (2016). SemEval 2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41, San Diego, California. Association for Computational Linguistics.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners.
Sakketou, F., Lahnala, A., Vogel, L., and Flek, L. (2022). Investigating user radicalization: A novel dataset for identifying fine-grained temporal shifts in opinion. In LREC-2022 proceedings, pages 3798–3808, Marseille, France. ELRA.
Tutek, M., Sekulić, I., Gombar, P., Paljak, I., Čulinović, F., Boltužić, F., Karan, M., Alagić, D., and Šnajder, J. (2016). TakeLab at SemEval-2016 task 6: Stance classification in tweets using a genetic algorithm based ensemble. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 464–468, San Diego, California. Association for Computational Linguistics.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, pages 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
Wang, F., Lan, M., and Wu, Y. (2017). ECNU at SemEval-2017 task 8: Rumour evaluation using effective features and supervised ensemble models. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 491–496, Vancouver, Canada. Association for Computational Linguistics.
Wei, W., Zhang, X., Liu, X., Chen, W., and Wang, T. (2016). pkudblab at SemEval-2016 task 6 : A specific convolutional neural network system for effective stance detection. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016), pages 384–388, San Diego, California. Association for Computational Linguistics.
Won, M. and Fernandes, J. (2022). SS-PT: A stance and sentiment data set from Portuguese quoted tweets. In PROPOR-2022 proceedings, pages 110–121, Fortaleza, Brazil. Springer.
Yang, R., Xie, W., Liu, C., and Yu, D. (2019). BLCU NLP at SemEval-2019 task 7: An inference chain-based GPT model for rumour evaluation. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1090–1096, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Yin, W., Hay, J., and Roth, D. (2019). Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach.
Zarrella, G. and Marsh, A. (2016). MITRE at SemEval-2016 task 6: Transfer learning for stance detection. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 458–463, San Diego, California. Association for Computational Linguistics.
Zhang, B., Ding, D., and Jing, L. (2023). How would Stance Detection Techniques Evolve after the Launch of ChatGPT?
Zhang, L., Wang, S., and Liu, B. (2018). Deep learning for sentiment analysis: A survey. WIREs Data Mining and Knowledge Discovery, 8(4):e1253.
Publicado
25/09/2023
Como Citar
DE FONSECA, Felipe Penhorate Carvalho; PARABONI, Ivandré; DIGIAMPIETRI, Luciano Antonio.
Contextual stance classification using prompt engineering. In: SIMPÓSIO BRASILEIRO DE TECNOLOGIA DA INFORMAÇÃO E DA LINGUAGEM HUMANA (STIL), 14. , 2023, Belo Horizonte/MG.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2023
.
p. 33-42.
DOI: https://doi.org/10.5753/stil.2023.233708.
