Performance in a dialectal profiling task of LLMs for varieties of Brazilian Portuguese
Resumo
Vieses de diferentes tipos são reproduzidos em respostas geradas por LLMs, inclusive dialetais. Um estudo baseado em engenharia de prompt foi realizado para descobrir como os LLMs discriminam as variedades do português brasileiro, especificamente se regras sociolinguísticas são consideradas por quatro LLMs – GPT 3.5, GPT-4o, Gemini e Sabiá-2 – na geração de suas respostas. Os resultados oferecem contribuições sociolinguísticas para uma tecnologia de PLN com equidade dialetal.
Referências
Abracado, J. and Martins, M. A. (2015). Mapeamento sociolinguístico do português brasileiro. Editora Contexto.
Baugh, J. (2018). Linguistics in pursuit of justice. Cambridge University Press.
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623.
Dacon, J. (2022). Towards a deep multi-layered dialectal language analysis: A case study of african-american english. arXiv preprint arXiv:2206.08978.
Dacon, J. and Tang, J. (2021). What truly matters? using linguistic cues for analyzing the# blacklivesmatter movement and its counter protests: 2013 to 2020. arXiv preprint arXiv:2109.12192.
Dacon, J., Liu, H., and Tang, J. (2022). Evaluating and mitigating inherent linguistic bias of african american english through inference. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1442–1454.
Danziger, S. (2018). Where intelligence lies: Externalist and sociolinguistic perspectives on the turing test and ai. In Philosophy and Theory of Artificial Intelligence 2017, pages 158–174. Springer.
Eckert, P. (2008). Variation and the indexical field. Journal of sociolinguistics, 12(4):453–476.
Fleisig, E., Smith, G., Bossi, M., Rustagi, I., Yin, X., and Klein, D. (2024). Linguistic bias in chatgpt: Language models reinforce dialect discrimination. arXiv preprint arXiv:2406.08818.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [link]
Freitag, R. M. K. (2016). Sociolinguística no/do Brasil. Cadernos de Estudos Linguísısticos, 58(3):445–460.
Freitag, R. M. K. (2019). Kappa statistic for judgment agreement in Sociolinguistics. Revista de Estudos da Linguagem, 27(4), 1591-1612.
Freitag, R. M. K. (2021). Preconceito linguístico para humanizar as máquinas. Cadernos de Linguística, 2(4):e495–e495.
Freitag, R. M. K., Severo, C. G., Rost-Snichelotto, C. A., and Tavares, M. A. (2015). Como o brasileiro acha que fala? desafios e propostas para a caracterização do "português brasileiro”. Signo y seña, (28):65–87.
Freitag, R. M. K., Severo, C. G., Rost-Snichelotto, C. A., and Tavares, M. A. (2016). Como os brasileiros acham que falam? percepções sociolinguísticas de universitários do sul e do nordeste. Todas as Letras, 18(2): 64-84.
Grieve, J., Bartl, S., Fuoli, M., Grafmiller, J., Huang, W., Jawerbaum, A., Murakami, A., Perlman, M., Roemling, D., and Winter, B. (2024). The sociolinguistic foundations of language modeling. arXiv preprint arXiv:2407.09241.
Liu, Y., Held, W., and Yang, D. (2023). Dada: Dialect adaptation via dynamic aggregation of linguistic rules. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13776–13793.
Mengesha, Z., Heldreth, C., Lahav, M., Sublewski, J., and Tuennerman, E. (2021). “I don’t think these devices are very culturally sensitive.”—impact of automated speech recognition errors on african americans. Frontiers in Artificial Intelligence, 4:725911.
Nee, J., Macfarlane Smith, G., Sheares, A., and Rustagi, I. (2021). Advancing social justice through linguistic justice: Strategies for building equity fluent nlp technology. In Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1–9.
Nee, J., Smith, G. M., Sheares, A., and Rustagi, I. (2022). Linguistic justice as a framework for designing, developing, and managing natural language processing tools. Big Data & Society, 9(1):20539517221090930. [link]
Preston, D. R. (2010). Language, people, salience, space: perceptual dialectology and language regard. Dialectologia: revista electrònica, pages 87–131.
Roncarati, C., Abracado, J., and Heye, J. B. (2003). Português brasileiro: contato linguístico, heterogeneidade e história, volume 2. 7 Letras.
Sales Almeida, T., Abonizio, H., Nogueira, R., and Pires, R. (2024). Sabiá-2: A new generation of portuguese large language models. arXiv e-prints, pages arXiv–2403.
Shrawgi, H., Rath, P., Singhal, T., and Dandapat, S. (2024). Uncovering stereotypes in large language models: A task complexity-based approach. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1841–1857.
Wolfram, W. and Eisenhauer, K. (2019). Implicit sociolinguistic bias and social justice. In The Routledge Companion to the Work of John R. Rickford, pages 269–280. Routledge.
Baugh, J. (2018). Linguistics in pursuit of justice. Cambridge University Press.
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623.
Dacon, J. (2022). Towards a deep multi-layered dialectal language analysis: A case study of african-american english. arXiv preprint arXiv:2206.08978.
Dacon, J. and Tang, J. (2021). What truly matters? using linguistic cues for analyzing the# blacklivesmatter movement and its counter protests: 2013 to 2020. arXiv preprint arXiv:2109.12192.
Dacon, J., Liu, H., and Tang, J. (2022). Evaluating and mitigating inherent linguistic bias of african american english through inference. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1442–1454.
Danziger, S. (2018). Where intelligence lies: Externalist and sociolinguistic perspectives on the turing test and ai. In Philosophy and Theory of Artificial Intelligence 2017, pages 158–174. Springer.
Eckert, P. (2008). Variation and the indexical field. Journal of sociolinguistics, 12(4):453–476.
Fleisig, E., Smith, G., Bossi, M., Rustagi, I., Yin, X., and Klein, D. (2024). Linguistic bias in chatgpt: Language models reinforce dialect discrimination. arXiv preprint arXiv:2406.08818.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [link]
Freitag, R. M. K. (2016). Sociolinguística no/do Brasil. Cadernos de Estudos Linguísısticos, 58(3):445–460.
Freitag, R. M. K. (2019). Kappa statistic for judgment agreement in Sociolinguistics. Revista de Estudos da Linguagem, 27(4), 1591-1612.
Freitag, R. M. K. (2021). Preconceito linguístico para humanizar as máquinas. Cadernos de Linguística, 2(4):e495–e495.
Freitag, R. M. K., Severo, C. G., Rost-Snichelotto, C. A., and Tavares, M. A. (2015). Como o brasileiro acha que fala? desafios e propostas para a caracterização do "português brasileiro”. Signo y seña, (28):65–87.
Freitag, R. M. K., Severo, C. G., Rost-Snichelotto, C. A., and Tavares, M. A. (2016). Como os brasileiros acham que falam? percepções sociolinguísticas de universitários do sul e do nordeste. Todas as Letras, 18(2): 64-84.
Grieve, J., Bartl, S., Fuoli, M., Grafmiller, J., Huang, W., Jawerbaum, A., Murakami, A., Perlman, M., Roemling, D., and Winter, B. (2024). The sociolinguistic foundations of language modeling. arXiv preprint arXiv:2407.09241.
Liu, Y., Held, W., and Yang, D. (2023). Dada: Dialect adaptation via dynamic aggregation of linguistic rules. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13776–13793.
Mengesha, Z., Heldreth, C., Lahav, M., Sublewski, J., and Tuennerman, E. (2021). “I don’t think these devices are very culturally sensitive.”—impact of automated speech recognition errors on african americans. Frontiers in Artificial Intelligence, 4:725911.
Nee, J., Macfarlane Smith, G., Sheares, A., and Rustagi, I. (2021). Advancing social justice through linguistic justice: Strategies for building equity fluent nlp technology. In Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1–9.
Nee, J., Smith, G. M., Sheares, A., and Rustagi, I. (2022). Linguistic justice as a framework for designing, developing, and managing natural language processing tools. Big Data & Society, 9(1):20539517221090930. [link]
Preston, D. R. (2010). Language, people, salience, space: perceptual dialectology and language regard. Dialectologia: revista electrònica, pages 87–131.
Roncarati, C., Abracado, J., and Heye, J. B. (2003). Português brasileiro: contato linguístico, heterogeneidade e história, volume 2. 7 Letras.
Sales Almeida, T., Abonizio, H., Nogueira, R., and Pires, R. (2024). Sabiá-2: A new generation of portuguese large language models. arXiv e-prints, pages arXiv–2403.
Shrawgi, H., Rath, P., Singhal, T., and Dandapat, S. (2024). Uncovering stereotypes in large language models: A task complexity-based approach. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1841–1857.
Wolfram, W. and Eisenhauer, K. (2019). Implicit sociolinguistic bias and social justice. In The Routledge Companion to the Work of John R. Rickford, pages 269–280. Routledge.
Publicado
17/11/2024
Como Citar
FREITAG, Raquel M. Ko; GOIS, Túlio Sousa de.
Performance in a dialectal profiling task of LLMs for varieties of Brazilian Portuguese. In: SIMPÓSIO BRASILEIRO DE TECNOLOGIA DA INFORMAÇÃO E DA LINGUAGEM HUMANA (STIL), 15. , 2024, Belém/PA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 317-326.
DOI: https://doi.org/10.5753/stil.2024.241891.
