Esquecida no Churrasco: Um método para validação e correção responsável de traduções automáticas do BBQ para Português do Brasil
Resumo
Com os rápidos avanços em Processamento de Linguagem Natural, torna-se necessário o desenvolvimento de ferramentas de avaliação de performances dos modelos linguísticos diante da destilação de vieses sociais. Neste artigo, então, apresentamos um método de adaptação de datasets que contempla adequações de fatores linguísticos e socioculturais do Brasil, uma demonstração de aplicação do método ao dataset BBQ e a descrição da implementação da nossa plataforma autoral de validação.Referências
Aryabumi, V. et al. (2024). Aya 23: Open weight releases to further multilingual progress. arXiv:2405.15032.
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In FAccT, pages 610–623.
Bonil, G., Hashiguti, S., Silva, J., Gondim, J., Maia, H., Silva, N., Pedrini, H., and Avila, S. (2025). Yet another algorithmic bias: A discursive analysis of large language models reinforcing dominant discourses on gender and race. arXiv:2508.10304.
Brown, T. B. et al. (2020). Language models are few-shot learners. arXiv:2005.14165.
Gadiraju, V., Kane, S., Dev, S., Taylor, A., Wang, D., Denton, R., and Brewer, R. (2023). ”i wouldn’t say offensive but...”: Disability-centered perspectives on large language models. In FAccT, page 205–216.
Huang, Y. and Xiong, D. (2023). Cbbq: A chinese bias benchmark dataset curated with human-ai collaboration for large language models. arXiv:2306.16244.
Ivetta, G., Palombini, P., Martinelli, S., Gomez, M. J., Dev, S., Prabhakaran, V., and Benotti, L. (2025). Adaptive data collection for latin-american community-sourced evaluation of stereotypes (laces). arXiv:2510.24958.
Jiang, A. Q. et al. (2024). Mixtral of experts. arXiv:2401.04088.
Jin, J., Kim, J., Lee, N., Yoo, H., Oh, A., and Lee, H. (2024). Kobbq: Korean bias benchmark for question answering. arXiv:2307.16778.
Messeri, L. and Crockett, M. J. (2024). Artificial intelligence and illusions of understanding in scientific research. Nature, 627:49–58.
Mitchell, M. et al. (2025). Shades: Towards a multilingual assessment of stereotypes in large language models. In NAACL, pages 11995–12041.
Moreira, D. A. B. et al. (2024). Fairpivara: Reducing and assessing biases in clip-based multimodal models. arXiv:2409.19474.
Nadeem, M., Bethke, A., and Reddy, S. (2020). Stereoset: Measuring stereotypical bias in pretrained language models. arXiv:2004.09456.
Nangia, N., Vania, C., Bhalerao, R., and Bowman, S. R. (2020). CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In EMNLP, pages 1953–1967.
Neplenbroek, V., Bisazza, A., and Fernández, R. (2024). Mbbq: A dataset for cross-lingual comparison of stereotypes in generative llms. arXiv:2406.07243.
OLMo, T. et al. (2025). 2 olmo 2 furious. arXiv:2501.00656.
OpenAI et al. (2024). Gpt-4 technical report. arXiv:2303.08774.
Ovalle, A., Goyal, P., Dhamala, J., Jaggers, Z., Chang, K.-W., Galstyan, A., Zemel, R., and Gupta, R. (2023). “i’m fully who i am”: Towards centering transgender and non-binary voices to measure biases in open language generation. In FAccT, page 1246–1266.
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., and Bowman, S. R. (2022). Bbq: A hand-built bias benchmark for question answering. arXiv:2110.08193.
Satheesh, S., Klug, K., Beckh, K., Allende-Cid, H., Houben, S., and Hassan, T. (2025). Gg-bbq: German gender bias benchmark for question answering. arXiv:2507.16410.
Shimabucoro, L., Ruder, S., Kreutzer, J., Fadaee, M., and Hooker, S. (2024). Llm see, llm do: Guiding data generation to target non-differentiable objectives. arXiv:2407.01490.
Silva, J. et al. (2024). Avaliação de ferramentas de Ética no levantamento de considerações Éticas de modelos de linguagem em português. In LAAI, pages 61–64. SBC.
Soares, T., Gumiel, Y., Junqueira, R., Gomes, T., and Pagano, A. (2023). Viés de gênero na tradução automática do gpt-3.5 turbo: avaliando o par linguístico inglês-português. In STIL, pages 167–176.
Talat, Z., Blix, H., Valvoda, J., Ganesh, M. I., Cotterell, R., and Williams, A. (2022). On the machine learning of ethical judgments from natural language. In NAACL, pages 769–779.
Team, G. et al. (2024). Gemma: Open models based on gemini research and technology. arXiv:2403.08295.
Touvron, H. et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288.
Zhao, J., Khashabi, D., Khot, T., Sabharwal, A., and Chang, K.-W. (2021). Ethical-advice taker: Do language models understand natural language interventions? arXiv:2106.01465.
Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In FAccT, pages 610–623.
Bonil, G., Hashiguti, S., Silva, J., Gondim, J., Maia, H., Silva, N., Pedrini, H., and Avila, S. (2025). Yet another algorithmic bias: A discursive analysis of large language models reinforcing dominant discourses on gender and race. arXiv:2508.10304.
Brown, T. B. et al. (2020). Language models are few-shot learners. arXiv:2005.14165.
Gadiraju, V., Kane, S., Dev, S., Taylor, A., Wang, D., Denton, R., and Brewer, R. (2023). ”i wouldn’t say offensive but...”: Disability-centered perspectives on large language models. In FAccT, page 205–216.
Huang, Y. and Xiong, D. (2023). Cbbq: A chinese bias benchmark dataset curated with human-ai collaboration for large language models. arXiv:2306.16244.
Ivetta, G., Palombini, P., Martinelli, S., Gomez, M. J., Dev, S., Prabhakaran, V., and Benotti, L. (2025). Adaptive data collection for latin-american community-sourced evaluation of stereotypes (laces). arXiv:2510.24958.
Jiang, A. Q. et al. (2024). Mixtral of experts. arXiv:2401.04088.
Jin, J., Kim, J., Lee, N., Yoo, H., Oh, A., and Lee, H. (2024). Kobbq: Korean bias benchmark for question answering. arXiv:2307.16778.
Messeri, L. and Crockett, M. J. (2024). Artificial intelligence and illusions of understanding in scientific research. Nature, 627:49–58.
Mitchell, M. et al. (2025). Shades: Towards a multilingual assessment of stereotypes in large language models. In NAACL, pages 11995–12041.
Moreira, D. A. B. et al. (2024). Fairpivara: Reducing and assessing biases in clip-based multimodal models. arXiv:2409.19474.
Nadeem, M., Bethke, A., and Reddy, S. (2020). Stereoset: Measuring stereotypical bias in pretrained language models. arXiv:2004.09456.
Nangia, N., Vania, C., Bhalerao, R., and Bowman, S. R. (2020). CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In EMNLP, pages 1953–1967.
Neplenbroek, V., Bisazza, A., and Fernández, R. (2024). Mbbq: A dataset for cross-lingual comparison of stereotypes in generative llms. arXiv:2406.07243.
OLMo, T. et al. (2025). 2 olmo 2 furious. arXiv:2501.00656.
OpenAI et al. (2024). Gpt-4 technical report. arXiv:2303.08774.
Ovalle, A., Goyal, P., Dhamala, J., Jaggers, Z., Chang, K.-W., Galstyan, A., Zemel, R., and Gupta, R. (2023). “i’m fully who i am”: Towards centering transgender and non-binary voices to measure biases in open language generation. In FAccT, page 1246–1266.
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., and Bowman, S. R. (2022). Bbq: A hand-built bias benchmark for question answering. arXiv:2110.08193.
Satheesh, S., Klug, K., Beckh, K., Allende-Cid, H., Houben, S., and Hassan, T. (2025). Gg-bbq: German gender bias benchmark for question answering. arXiv:2507.16410.
Shimabucoro, L., Ruder, S., Kreutzer, J., Fadaee, M., and Hooker, S. (2024). Llm see, llm do: Guiding data generation to target non-differentiable objectives. arXiv:2407.01490.
Silva, J. et al. (2024). Avaliação de ferramentas de Ética no levantamento de considerações Éticas de modelos de linguagem em português. In LAAI, pages 61–64. SBC.
Soares, T., Gumiel, Y., Junqueira, R., Gomes, T., and Pagano, A. (2023). Viés de gênero na tradução automática do gpt-3.5 turbo: avaliando o par linguístico inglês-português. In STIL, pages 167–176.
Talat, Z., Blix, H., Valvoda, J., Ganesh, M. I., Cotterell, R., and Williams, A. (2022). On the machine learning of ethical judgments from natural language. In NAACL, pages 769–779.
Team, G. et al. (2024). Gemma: Open models based on gemini research and technology. arXiv:2403.08295.
Touvron, H. et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288.
Zhao, J., Khashabi, D., Khot, T., Sabharwal, A., and Chang, K.-W. (2021). Ethical-advice taker: Do language models understand natural language interventions? arXiv:2106.01465.
Publicado
19/10/2026
Como Citar
CARDOSO, Luiza Rodrigues; RIBEIRO, Leo Sampaio Ferraz.
Esquecida no Churrasco: Um método para validação e correção responsável de traduções automáticas do BBQ para Português do Brasil. In: SIMPÓSIO BRASILEIRO DE TECNOLOGIA DA INFORMAÇÃO E DA LINGUAGEM HUMANA (STIL), 17. , 2026, Cuiabá/MT.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 610-615.
DOI: https://doi.org/10.5753/stil.2026.26812.
