Avaliação do impacto de entradas multimodais em LLMs: um estudo de caso de respostas ao POSCOMP
Resumo
Large Language Models (LLMs) podem apoiar alunos no estudo e docentes na criação de provas devido suportarem entradas multimodais. Porém, ainda há uma lacuna na literatura sobre a avaliação da multimodalidade dos LLMs na resolução de questões de provas. A literatura apresenta uma estrutura experimental que não abrange os possíveis tipos de entrada. Este estudo visa preencher tal lacuna, avaliando o efeito de imagens, PDFs e textos em LATEX nas acurácias do Gemini 1.5 Pro do Google e ChatGPT-4o da OpenAI via API e interface web em questões de LFA do POSCOMP. Destarte, observou-se que o Gemini superou o ChatGPT em LFA, que questões pictóricas têm menor acurácia e que a API potencializa a vantagem do LATEX. Em suma, os achados de pesquisa têm o potencial de impactar alunos, docentes e o setor produtivo.Referências
Abu-Haifa, M., Etawi, B., Alkhatatbeh, H., and Ababneh, A. (2024). Comparative analysis of chatgpt, gpt-4, and microsoft copilot chatbots for gre test. International Journal of Learning, Teaching and Educational Research, 23:327–347.
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. (2024). Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
Dao, X.-Q., Le, N.-B., Phan, X.-D., and Ngo, B.-B. (2023). Can chatgpt pass the vietnamese national high school graduation examination? arXiv preprint arXiv:2306.09170.
Donner, C. G. G. (2024). Misinformation detection methods using large language models and evaluation of application programming interfaces. Master’s thesis, University of Oklahoma.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
Martínez-Plumed, F., Contreras-Ochando, L., Ferri, C., Hernández-Orallo, J., Kull, M., Lachiche, N., Ramírez-Quintana, M. J., and Flach, P. (2021). Crisp-dm twenty years later: From data mining processes to data science trajectories. IEEE Transactions on Knowledge and Data Engineering, 33(8):3048–3061.
Mendonça, N. C. (2024). Evaluating chatgpt-4 vision on brazil’s national undergraduate computer science exam. ACM Trans. Comput. Educ., 24(3).
Mickle, T. (2023). Big tech rebounds and preps for transformative a.i. investments. [link]. Acesso em: 8 ago 2024.
Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., and Ioannidis, J. (2017). A manifesto for reproducible science. Nature human behaviour, 1(1):1–9.
Plevris, V., Papazafeiropoulos, G., and Rios, A. J. (2023). Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of chatgpt-3.5, chatgpt-4, and google bard.
Raihan, N., Siddiq, M. L., Santos, J. C., and Zampieri, M. (2025). Large language models in computer science education: A systematic literature review. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, pages 938–944.
Saldanha, M. S. and Digiampietri, L. A. (2024). Chatgpt and bard performance on the poscomp exam. In Proceedings of the 20th Brazilian Symposium on Information Systems, SBSI ’24, New York, NY, USA. Association for Computing Machinery.
SBC, S. B. d. C. (2024). Exame Nacional para Ingresso na Pós-Graduação em Computação (POSCOMP). [link]. Acesso em: 16 jul. 2024.
Viegas, C. V. (2024). Avaliando a capacidade de llms na resolução de questões do poscomp. Repositório Institucional da UFCG. [link].
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
Williams, S. and Huckle, J. (2024). Easy problems that llms get wrong. [link].
Wu, S., Xiong, Y., Cui, Y., Wu, H., Chen, C., Yuan, Y., Huang, L., Liu, X., Kuo, T.-W., Guan, N., et al. (2024). Retrieval-augmented generation for natural language processing: A survey. arXiv preprint arXiv:2407.13193.
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., and He, L. (2022). A survey of human-in-the-loop for machine learning. Future Generation Computer Systems, 135:364–381.
Zhang, P. and Tur, G. (2024). A systematic review of chatgpt use in k-12 education. European Journal of Education, 59(2):e12599.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223.
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. (2024). Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
Dao, X.-Q., Le, N.-B., Phan, X.-D., and Ngo, B.-B. (2023). Can chatgpt pass the vietnamese national high school graduation examination? arXiv preprint arXiv:2306.09170.
Donner, C. G. G. (2024). Misinformation detection methods using large language models and evaluation of application programming interfaces. Master’s thesis, University of Oklahoma.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
Martínez-Plumed, F., Contreras-Ochando, L., Ferri, C., Hernández-Orallo, J., Kull, M., Lachiche, N., Ramírez-Quintana, M. J., and Flach, P. (2021). Crisp-dm twenty years later: From data mining processes to data science trajectories. IEEE Transactions on Knowledge and Data Engineering, 33(8):3048–3061.
Mendonça, N. C. (2024). Evaluating chatgpt-4 vision on brazil’s national undergraduate computer science exam. ACM Trans. Comput. Educ., 24(3).
Mickle, T. (2023). Big tech rebounds and preps for transformative a.i. investments. [link]. Acesso em: 8 ago 2024.
Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., and Ioannidis, J. (2017). A manifesto for reproducible science. Nature human behaviour, 1(1):1–9.
Plevris, V., Papazafeiropoulos, G., and Rios, A. J. (2023). Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of chatgpt-3.5, chatgpt-4, and google bard.
Raihan, N., Siddiq, M. L., Santos, J. C., and Zampieri, M. (2025). Large language models in computer science education: A systematic literature review. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, pages 938–944.
Saldanha, M. S. and Digiampietri, L. A. (2024). Chatgpt and bard performance on the poscomp exam. In Proceedings of the 20th Brazilian Symposium on Information Systems, SBSI ’24, New York, NY, USA. Association for Computing Machinery.
SBC, S. B. d. C. (2024). Exame Nacional para Ingresso na Pós-Graduação em Computação (POSCOMP). [link]. Acesso em: 16 jul. 2024.
Viegas, C. V. (2024). Avaliando a capacidade de llms na resolução de questões do poscomp. Repositório Institucional da UFCG. [link].
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
Williams, S. and Huckle, J. (2024). Easy problems that llms get wrong. [link].
Wu, S., Xiong, Y., Cui, Y., Wu, H., Chen, C., Yuan, Y., Huang, L., Liu, X., Kuo, T.-W., Guan, N., et al. (2024). Retrieval-augmented generation for natural language processing: A survey. arXiv preprint arXiv:2407.13193.
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., and He, L. (2022). A survey of human-in-the-loop for machine learning. Future Generation Computer Systems, 135:364–381.
Zhang, P. and Tur, G. (2024). A systematic review of chatgpt use in k-12 education. European Journal of Education, 59(2):e12599.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223.
Publicado
20/07/2025
Como Citar
CARRETEIRO, Florindo R. S.; SOUSA, Fernando A. de; MARCACINI, Ricardo M.; LOBATO, Fábio M. F..
Avaliação do impacto de entradas multimodais em LLMs: um estudo de caso de respostas ao POSCOMP. In: WORKSHOP SOBRE EDUCAÇÃO EM COMPUTAÇÃO (WEI), 33. , 2025, Maceió/AL.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 678-689.
ISSN 2595-6175.
DOI: https://doi.org/10.5753/wei.2025.8455.
