Evaluating Large Language Models through Multidimensional Item Response Theory: A Comprehensive Case Study on ENEM - Taschetto, Leonardo; Fileto, Renato (Anais do Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL); 2025: Anais do XVI Simpósio Brasileiro de Tecnologia da Informação e da Linguagem HumanaDO - 10.5753/stil.2025.37846, 2025)
LLM evaluations on tasks like high-stakes multidisciplinary tests still rely on raw accuracy, a metric that weights easy and difficult questions equally and ignores guessing. To help bridge this methodological gap, we repurpose the official three-parameter logistic Item Response Theory (IRT) calibration that the Brazilian education authority (INEP) uses to score humans on the Exame Nacional do Ensino Médio (ENEM), and apply it to LLM responses. We then fit a four-dimensional 3-PL model aligned with ENEM’s knowledge domains. Results show that similar accuracies can mask proficiency gaps exceeding one standard deviation across domains. Mathematics remains the toughest domain for both humans and models, whereas questions on Human Sciences are systematically easier for both.