Avaliação de LLMs Multimodais na Modalidade Programação da Olimpíada Brasileira de Informática: Uma Arquitetura Experimental
Resumo
Este artigo tem como objetivo desenvolver e avaliar uma arquitetura para medir a capacidade de Grandes Modelos de Linguagem Multimodais (MLLMs) na resolução de desafios algorítmicos complexos na Modalidade Programação da Olimpíada Brasileira de Informática (OBI). Para isso, foi construída uma infraestrutura automatizada e experimental composta por um agente gerador de código baseado em LLM e um avaliador (Judge) para validação dos casos de teste. Dessa forma, um orquestrador gerencia o fluxo entre o banco de problemas, a estruturação de templates de prompts e os serviços externos (LLMs). Para simplificação dos resultados, adotou-se a seguinte configuração: o código gerado em Python e a técnica de prompt zero-shot, comparando-se enunciados contendo imagens no formato Base64 com os enunciados puramente textuais. O estudo traz um banco de questões ainda não explorado por MLLMs, além de traçar um paralelo pertinente entre a programação competitiva e a Engenharia de Software, no qual a interpretação de enunciados assemelha-se à modelagem de requisitos de sistemas e a validação de corner cases reflete a prática de testes unitários. Os resultados demonstram que o impacto da inclusão de imagens varia significativamente entre as ferramentas; embora o contexto visual possa adicionar ruído ou complexidade em alguns cenários, a análise comparativa revelou uma melhora no desempenho em problemas avançados de grafos e programação dinâmica em 3 dos 5 modelos testados, evidenciando que o suporte multimodal atua como um fator relevante para a resolução correta de desafios de alta complexidade algorítmica.
Palavras-chave:
Modelos de Linguagem Multimodais, Geração Automática de Código, Engenharia de Software Experimental, Teste de Software, Programação Competitiva
Referências
Thales Sales Almeida, Thiago Laitz, Giovana Kerche Bonás, and Rodrigo Nogueira. 2023. BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams. arXiv preprint arXiv:2307.05410 abs/2307.05410 (2023). [link]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33. 1877–1901.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond’e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo Bavarian, Clemens Winter, Phil Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv.org abs/2107.03374 (July 2021), 29 pages. DOI: 10.48550/arXiv.2107.03374
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394 abs/2306.13394 (2023). [link]
GEMA-LAB. 2026. Problems of the OBI Dataset. [link] Repositório GitHub.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020). [link]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. ArXiv abs/2304.08485 (2023). [link]
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun yue Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. In International Conference on Learning Representations (ICLR). [link]
Nabor C. Mendonça. 2024. Evaluating ChatGPT-4 Vision on Brazil’s National Undergraduate Computer Science Exam. ACM Transactions on Computing Education 24 (2024), 1 – 56. [link]
Isela Mendoza, Fernando Silva Filho, Gustavo Medeiros, Aline Paes, and Vânia O. Neves. 2024. Comparative Analysis of Large LanguageModel Tools for Automated Test Data Generation from BDD. In Anais do XXXVIII Simpósio Brasileiro de Engenharia de Software (SBES). Sociedade Brasileira de Computação - SBC, Porto Alegre, RS, Brasil, 280–290. DOI: 10.5753/sbes.2024.3423
Desnes Nunes, Ricardo Primi, Ramon Pires, Roberto Lotufo, and RodrigoNogueira. 2023. Evaluating GPT-3.5 and GPT-4 models on Brazilian university admission exams. arXiv preprint arXiv:2303.17003 (2023).
Olimpíada Brasileira de Informática. 2025. Provas e Gabaritos de Edições Passadas. [link] Acessado em: 27 abr. 2026.
Esdras Caleb Oliveira Silva, Roberta de Souza Coelho, and Lyrene Fernandes da Silva. 2025. LLMs as Test Generators: A Comparative Benchmarking Study. In Anais do XXXIX Simpósio Brasileiro de Engenharia de Software (SBES). Sociedade Brasileira de Computação - SBC, Porto Alegre, RS, Brasil, 25–36. DOI: 10.5753/, sbes.2025.9618
Cayo Viegas, Rohit Gheyi, and Márcio Ribeiro. 2025. Assessing the Capability of LLMs in Solving POSCOMP Questions. arXiv preprint arXiv:2505.20338 abs/2505.20338 (2025). [link]
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations. [link]
John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Adriano Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida Wang, and Ofir Press. 2024. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? arXiv preprint arXiv:2410.03859 abs/2410.03859 (2024). [link]
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023), 9556–9567. [link]
Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hang Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang, Wenhao Chai, Aleksandra Korolova, Peter Henderson, Sanjeev Arora, Pramod Viswanath, Jingbo Shang, and Saining Xie. 2025. LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? arXiv preprint arXiv:2506.11928 abs/2506.11928 (2025). [link]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33. 1877–1901.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond’e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo Bavarian, Clemens Winter, Phil Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv.org abs/2107.03374 (July 2021), 29 pages. DOI: 10.48550/arXiv.2107.03374
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394 abs/2306.13394 (2023). [link]
GEMA-LAB. 2026. Problems of the OBI Dataset. [link] Repositório GitHub.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020). [link]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. ArXiv abs/2304.08485 (2023). [link]
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun yue Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. In International Conference on Learning Representations (ICLR). [link]
Nabor C. Mendonça. 2024. Evaluating ChatGPT-4 Vision on Brazil’s National Undergraduate Computer Science Exam. ACM Transactions on Computing Education 24 (2024), 1 – 56. [link]
Isela Mendoza, Fernando Silva Filho, Gustavo Medeiros, Aline Paes, and Vânia O. Neves. 2024. Comparative Analysis of Large LanguageModel Tools for Automated Test Data Generation from BDD. In Anais do XXXVIII Simpósio Brasileiro de Engenharia de Software (SBES). Sociedade Brasileira de Computação - SBC, Porto Alegre, RS, Brasil, 280–290. DOI: 10.5753/sbes.2024.3423
Desnes Nunes, Ricardo Primi, Ramon Pires, Roberto Lotufo, and RodrigoNogueira. 2023. Evaluating GPT-3.5 and GPT-4 models on Brazilian university admission exams. arXiv preprint arXiv:2303.17003 (2023).
Olimpíada Brasileira de Informática. 2025. Provas e Gabaritos de Edições Passadas. [link] Acessado em: 27 abr. 2026.
Esdras Caleb Oliveira Silva, Roberta de Souza Coelho, and Lyrene Fernandes da Silva. 2025. LLMs as Test Generators: A Comparative Benchmarking Study. In Anais do XXXIX Simpósio Brasileiro de Engenharia de Software (SBES). Sociedade Brasileira de Computação - SBC, Porto Alegre, RS, Brasil, 25–36. DOI: 10.5753/, sbes.2025.9618
Cayo Viegas, Rohit Gheyi, and Márcio Ribeiro. 2025. Assessing the Capability of LLMs in Solving POSCOMP Questions. arXiv preprint arXiv:2505.20338 abs/2505.20338 (2025). [link]
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations. [link]
John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Adriano Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida Wang, and Ofir Press. 2024. SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? arXiv preprint arXiv:2410.03859 abs/2410.03859 (2024). [link]
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023), 9556–9567. [link]
Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hang Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang, Wenhao Chai, Aleksandra Korolova, Peter Henderson, Sanjeev Arora, Pramod Viswanath, Jingbo Shang, and Saining Xie. 2025. LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? arXiv preprint arXiv:2506.11928 abs/2506.11928 (2025). [link]
Publicado
11/09/2026
Como Citar
ÂNGELO, Victor Hugo S.; ANCHIETA, Lucas Heron S.; NASCIMENTO, Matheus Ryan S.; FERNANDES, Ricardo Vinicius A.; COSTA, Evandro de B..
Avaliação de LLMs Multimodais na Modalidade Programação da Olimpíada Brasileira de Informática: Uma Arquitetura Experimental. In: WORKSHOP BRASILEIRO DE ENGENHARIA DE SOFTWARE INTELIGENTE (ISE), 5. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 141-144.
DOI: https://doi.org/10.5753/ise.2026.32990.
