Performance in a dialectal profiling task of LLMs for varieties of Brazilian Portuguese
Abstract
Different of biases are reproduced in LLM-generated responses, including dialectal biases. A study based on prompt engineering was carried out to uncover how LLMs discriminate varieties of Brazilian Portuguese, specifically if sociolinguistic rules are taken into account in four LLMs: GPT 3.5, GPT-4o, Gemini, and Sabiá-2. The results offer sociolinguistic contributions for an equity fluent NLP technology.
References
Abracado, J. and Martins, M. A. (2015). Mapeamento sociolinguístico do português brasileiro. Editora Contexto.
Baugh, J. (2018). Linguistics in pursuit of justice. Cambridge University Press.
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623.
Dacon, J. (2022). Towards a deep multi-layered dialectal language analysis: A case study of african-american english. arXiv preprint arXiv:2206.08978.
Dacon, J. and Tang, J. (2021). What truly matters? using linguistic cues for analyzing the# blacklivesmatter movement and its counter protests: 2013 to 2020. arXiv preprint arXiv:2109.12192.
Dacon, J., Liu, H., and Tang, J. (2022). Evaluating and mitigating inherent linguistic bias of african american english through inference. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1442–1454.
Danziger, S. (2018). Where intelligence lies: Externalist and sociolinguistic perspectives on the turing test and ai. In Philosophy and Theory of Artificial Intelligence 2017, pages 158–174. Springer.
Eckert, P. (2008). Variation and the indexical field. Journal of sociolinguistics, 12(4):453–476.
Fleisig, E., Smith, G., Bossi, M., Rustagi, I., Yin, X., and Klein, D. (2024). Linguistic bias in chatgpt: Language models reinforce dialect discrimination. arXiv preprint arXiv:2406.08818.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [link]
Freitag, R. M. K. (2016). Sociolinguística no/do Brasil. Cadernos de Estudos Linguísısticos, 58(3):445–460.
Freitag, R. M. K. (2019). Kappa statistic for judgment agreement in Sociolinguistics. Revista de Estudos da Linguagem, 27(4), 1591-1612.
Freitag, R. M. K. (2021). Preconceito linguístico para humanizar as máquinas. Cadernos de Linguística, 2(4):e495–e495.
Freitag, R. M. K., Severo, C. G., Rost-Snichelotto, C. A., and Tavares, M. A. (2015). Como o brasileiro acha que fala? desafios e propostas para a caracterização do "português brasileiro”. Signo y seña, (28):65–87.
Freitag, R. M. K., Severo, C. G., Rost-Snichelotto, C. A., and Tavares, M. A. (2016). Como os brasileiros acham que falam? percepções sociolinguísticas de universitários do sul e do nordeste. Todas as Letras, 18(2): 64-84.
Grieve, J., Bartl, S., Fuoli, M., Grafmiller, J., Huang, W., Jawerbaum, A., Murakami, A., Perlman, M., Roemling, D., and Winter, B. (2024). The sociolinguistic foundations of language modeling. arXiv preprint arXiv:2407.09241.
Liu, Y., Held, W., and Yang, D. (2023). Dada: Dialect adaptation via dynamic aggregation of linguistic rules. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13776–13793.
Mengesha, Z., Heldreth, C., Lahav, M., Sublewski, J., and Tuennerman, E. (2021). “I don’t think these devices are very culturally sensitive.”—impact of automated speech recognition errors on african americans. Frontiers in Artificial Intelligence, 4:725911.
Nee, J., Macfarlane Smith, G., Sheares, A., and Rustagi, I. (2021). Advancing social justice through linguistic justice: Strategies for building equity fluent nlp technology. In Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1–9.
Nee, J., Smith, G. M., Sheares, A., and Rustagi, I. (2022). Linguistic justice as a framework for designing, developing, and managing natural language processing tools. Big Data & Society, 9(1):20539517221090930. [link]
Preston, D. R. (2010). Language, people, salience, space: perceptual dialectology and language regard. Dialectologia: revista electrònica, pages 87–131.
Roncarati, C., Abracado, J., and Heye, J. B. (2003). Português brasileiro: contato linguístico, heterogeneidade e história, volume 2. 7 Letras.
Sales Almeida, T., Abonizio, H., Nogueira, R., and Pires, R. (2024). Sabiá-2: A new generation of portuguese large language models. arXiv e-prints, pages arXiv–2403.
Shrawgi, H., Rath, P., Singhal, T., and Dandapat, S. (2024). Uncovering stereotypes in large language models: A task complexity-based approach. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1841–1857.
Wolfram, W. and Eisenhauer, K. (2019). Implicit sociolinguistic bias and social justice. In The Routledge Companion to the Work of John R. Rickford, pages 269–280. Routledge.
Baugh, J. (2018). Linguistics in pursuit of justice. Cambridge University Press.
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623.
Dacon, J. (2022). Towards a deep multi-layered dialectal language analysis: A case study of african-american english. arXiv preprint arXiv:2206.08978.
Dacon, J. and Tang, J. (2021). What truly matters? using linguistic cues for analyzing the# blacklivesmatter movement and its counter protests: 2013 to 2020. arXiv preprint arXiv:2109.12192.
Dacon, J., Liu, H., and Tang, J. (2022). Evaluating and mitigating inherent linguistic bias of african american english through inference. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1442–1454.
Danziger, S. (2018). Where intelligence lies: Externalist and sociolinguistic perspectives on the turing test and ai. In Philosophy and Theory of Artificial Intelligence 2017, pages 158–174. Springer.
Eckert, P. (2008). Variation and the indexical field. Journal of sociolinguistics, 12(4):453–476.
Fleisig, E., Smith, G., Bossi, M., Rustagi, I., Yin, X., and Klein, D. (2024). Linguistic bias in chatgpt: Language models reinforce dialect discrimination. arXiv preprint arXiv:2406.08818.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [link]
Freitag, R. M. K. (2016). Sociolinguística no/do Brasil. Cadernos de Estudos Linguísısticos, 58(3):445–460.
Freitag, R. M. K. (2019). Kappa statistic for judgment agreement in Sociolinguistics. Revista de Estudos da Linguagem, 27(4), 1591-1612.
Freitag, R. M. K. (2021). Preconceito linguístico para humanizar as máquinas. Cadernos de Linguística, 2(4):e495–e495.
Freitag, R. M. K., Severo, C. G., Rost-Snichelotto, C. A., and Tavares, M. A. (2015). Como o brasileiro acha que fala? desafios e propostas para a caracterização do "português brasileiro”. Signo y seña, (28):65–87.
Freitag, R. M. K., Severo, C. G., Rost-Snichelotto, C. A., and Tavares, M. A. (2016). Como os brasileiros acham que falam? percepções sociolinguísticas de universitários do sul e do nordeste. Todas as Letras, 18(2): 64-84.
Grieve, J., Bartl, S., Fuoli, M., Grafmiller, J., Huang, W., Jawerbaum, A., Murakami, A., Perlman, M., Roemling, D., and Winter, B. (2024). The sociolinguistic foundations of language modeling. arXiv preprint arXiv:2407.09241.
Liu, Y., Held, W., and Yang, D. (2023). Dada: Dialect adaptation via dynamic aggregation of linguistic rules. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13776–13793.
Mengesha, Z., Heldreth, C., Lahav, M., Sublewski, J., and Tuennerman, E. (2021). “I don’t think these devices are very culturally sensitive.”—impact of automated speech recognition errors on african americans. Frontiers in Artificial Intelligence, 4:725911.
Nee, J., Macfarlane Smith, G., Sheares, A., and Rustagi, I. (2021). Advancing social justice through linguistic justice: Strategies for building equity fluent nlp technology. In Proceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1–9.
Nee, J., Smith, G. M., Sheares, A., and Rustagi, I. (2022). Linguistic justice as a framework for designing, developing, and managing natural language processing tools. Big Data & Society, 9(1):20539517221090930. [link]
Preston, D. R. (2010). Language, people, salience, space: perceptual dialectology and language regard. Dialectologia: revista electrònica, pages 87–131.
Roncarati, C., Abracado, J., and Heye, J. B. (2003). Português brasileiro: contato linguístico, heterogeneidade e história, volume 2. 7 Letras.
Sales Almeida, T., Abonizio, H., Nogueira, R., and Pires, R. (2024). Sabiá-2: A new generation of portuguese large language models. arXiv e-prints, pages arXiv–2403.
Shrawgi, H., Rath, P., Singhal, T., and Dandapat, S. (2024). Uncovering stereotypes in large language models: A task complexity-based approach. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1841–1857.
Wolfram, W. and Eisenhauer, K. (2019). Implicit sociolinguistic bias and social justice. In The Routledge Companion to the Work of John R. Rickford, pages 269–280. Routledge.
Published
2024-11-17
How to Cite
FREITAG, Raquel M. Ko; GOIS, Túlio Sousa de.
Performance in a dialectal profiling task of LLMs for varieties of Brazilian Portuguese. In: BRAZILIAN SYMPOSIUM IN INFORMATION AND HUMAN LANGUAGE TECHNOLOGY (STIL), 15. , 2024, Belém/PA.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 317-326.
DOI: https://doi.org/10.5753/stil.2024.241891.
