EnsinaAI: Geração de Questões com Qualidade Pedagógica Auditada por Catálogos Declarativos
Resumo
Large Language Models (LLMs) viabilizam a Geração Automática de Itens em escala, mas tipicamente privilegiam fluência linguística em detrimento da fundamentação pedagógica. Apresentamos um framework que formaliza qualidade pedagógica como 61 regras declarativas em cinco catálogos (Evidence-Centered Design, Item-Writing Flaws — IWF, qualidade linguística, correção técnica e critérios de rubrica) executadas por um ciclo multiagente geração–auditoria–regeneração com motor de decisão determinístico. Em um estudo emergente, 150 questões de Redes de Computadores geradas por cinco LLMs distintos foram auditadas pelos catálogos linguístico (LIN) e IWF e atingiram 100% de conformidade com os critérios do motor (aprovação necessária, não suficiente, para a qualidade pedagógica), com aprovação em primeiro ciclo entre 40% e 70% (Qui-Quadrado, p ≈ 0,13, sem evidência de dependência do modelo ao nível α = 0,05). O estudo revelou ainda uma assimetria sistemática de retenção (IWF ≥ LIN) do GPT-4o-mini como árbitro, sugestiva de viés, mas ainda não distinguível da maior detectabilidade intrínseca das falhas de Item-Writing Flaws — padrão persistente mesmo em geradores sem relação com ele.
Palavras-chave:
Geração Automática de Itens, Qualidade Pedagógica, Modelos de Linguagem
Referências
Aguiar, R. K. B. d., Silva, T. R. d., Lima, R. V. d., Grosmann, D., Lima, B. V. A. d., and Oliveira, F. B. M. d. (2025). Métodos de ensino e aprendizagem em redes de computadores -uma revisão sistemática da literatura. RENOTE, 23(1):161–171.
Almond, R., Mislevy, R., and Steinberg, L. (2003). On the structure of educational assessments. Measurement: Interdisciplinary research and perspectives, 1(1):3–62.
Anderson, L. W. and Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy of educational objectives. Addison Wesley Longman, New York, complete edition edition.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., and Kaplan, J. (2022). Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
Behrens, J. T., Mislevy, R. J., DiCerbo, K. E., and Levy, R. (2010). An evidence centered design for learning and assessment in the digital world. CRESST Report 778, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Los Angeles, CA.
Brookhart, S. M. (2018). Appropriate criteria: Key to effective rubrics. Frontiers in Education, Volume 3 - 2018:22.
Caporal, A., Prado, M. R., Bini, I., and Boller, C. (2018). Padronização da correção de questões dissertativas para professores de saúde coletiva do curso de medicina em uma instituição de ensino superior do oeste do paraná. Revista Meta: Avaliação, 10(28):54–74.
Carvalho, F. S. and Soares, C. d. S. N. (2024). Avaliação por rubricas seguindo a taxonomia de bloom no diagnóstico de aprendizagem em algoritmos. In Anais do Simpósio Brasileiro de Informática na Educação.
Chan, K. W., Ali, F., Park, J., Sham, K. S. B., Tan, E. Y. T., Chong, F. W. C., Qian, K., and Sze, G. K. (2025). Automatic item generation in various stem subjects using large language model prompting. Computers and Education: Artificial Intelligence, 8:100344.
Cohn, C., Mohammed, N., Biswas, G., et al. (2025). Cotal: Human-in-the-loop prompt engineering for generalizable formative assessment scoring. arXiv preprint arXiv:2504.02323.
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., Bogart, C., Keylor, E., Kultur, C., Savelka, J., and Sakr, M. (2024). A comparative study of ai-generated (gpt-4) and human-crafted mcqs in programming education. In Proceedings of the 26th Australasian Computing Education Conference, ACE '24, pages 114–123, New York, NY, USA. Association for Computing Machinery.
Elkins, S., Kochmar, E., Cheung, J. C. K., and Serban, I. (2024). How teachers can use large language models and bloom's taxonomy to create educational quizzes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23084–23091.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv e-prints, page arXiv:2312.10997.
Gierl, M. J. and Haladyna, T. M., editors (2013). Automatic Item Generation: Theory and Practice. Routledge, New York, NY.
Gorgun, G. and Bulut, O. (2025). Instruction-tuned large-language models for quality control in automatic item generation: A feasibility study. Educational Measurement: Issues and Practice, 44(1):96–107.
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., and Fritz, M. (2023). Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, AISec '23, pages 79–90, New York, NY, USA. Association for Computing Machinery.
Grosmann, D., de Aguiar, R. K. B., da Silva, T. R., and Teixeira, M. M. (2025). Challenges, benefits, and perspectives on the use of virtual laboratories in computer networks teaching: A systematic mapping. In Sánchez, J., editor, Nuevas Ideas en Informática Educativa, volume 19, pages 87–97, Santiago de Chile.
Haladyna, T. M. (2004). Developing and Validating Multiple-Choice Test Items. Lawrence Erlbaum Associates, Inc., Mahwah, New Jersey, 3rd edition.
Haladyna, T. M. and Downing, S. M. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1):17–27.
Haladyna, T. M., Downing, S. M., and Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3):309–333.
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., and Liu, T. (2025). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2).
Jesus, G. R. d., Rêgo, R. M. d. L., and Souza, V. V. d. (2020). A questão da validade na avaliação educacional brasileira. Ensaio: Avaliação e Políticas Públicas em Educação.
Jia, R., Zhang, M., Liu, F., Jiang, B., Kuang, K., and Dai, Z. (2025). Eduagentqg: A multi-agent workflow framework for personalized question generation. arXiv preprint arXiv:2511.11635.
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. (2022). Language models (mostly) know what they know.
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. (2020). Dense passage retrieval for open-domain question answering. arXiv e-prints, page arXiv:2004.04906.
Kubsch, M., Grimm, A., Neumann, K., Drachsler, H., and Rummel, N. (2024). Using evidence-centered design to develop an automated system for tracking students physics learning in a digital learning environment. In Zhai, X. and Krajcik, J., editors, Uses of Artificial Intelligence in STEM Education, chapter 11, pages 230–249. Oxford University Press.
Kurdi, G., Leo, J., Parsia, B., Sattler, U., and Al-Emari, S. (2020). A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30(1):121–204.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems (NeurIPS), pages 9459–9474. Curran Associates, Inc.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. (2023). Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems (NeurIPS), pages 46534–46594.
Maus, H., Tschisgale, P., Kieser, F., Petersen, S., and Wulff, P. (2025). Developing and evaluating a large language model-based automated feedback system grounded in evidence-centered design for supporting physics problem solving. IEEE Transactions on Learning Technologies.
Mislevy, R. J., Behrens, J. T., DiCerbo, K. E., and Levy, R. (2012). Design and discovery in educational assessment: Evidence-centered design, psychometrics, and educational data mining. Journal of Educational Data Mining, 4(1):11–48.
Mohammed, O. E. A. H., Srinivasamurthy, S. K., Bhat, R., Eltahir, Y. A. M. A., Elshamly, H. A. H., Mohammed, F., and Hamad, B. (2025). Postgraduate clinical residency: The impact of multiple-choice question quality on exam success rates. Advances in Medical Education and Practice, 16:1381–1397. PMID: 40786957.
Moore, S., Costello, E., Nguyen, H. A., and Stamper, J. (2024). An automatic question usability evaluation toolkit. In Olney, A. M., Chounta, I.-A., Liu, Z., Santos, O. C., and Bittencourt, I. I., editors, Artificial Intelligence in Education, pages 31–46, Cham. Springer Nature Switzerland.
Moore, S., Nguyen, H. A., Chen, T., and Stamper, J. (2023). Assessing the quality of multiple-choice questions using gpt-4 and rule-based methods. In Viberg, O., Jivet, I., Muñoz-Merino, P. J., Perifanou, M., and Papathoma, T., editors, Responsive and Sustainable Educational Futures, volume 14200, pages 229–245, Cham. Springer Nature Switzerland.
Popham, W. J. (1997). What's wrong and what's right-with rubrics. Educational leadership, 55(2):72–75.
Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. In Inui, K., Jiang, J., Ng, V., and Wan, X., editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
Schmucker, R. and Moore, S. (2025). The impact of item-writing flaws on difficulty and discrimination in item response theory. arXiv preprint arXiv:2503.10533.
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. arXiv e-prints, page arXiv:2303.11366.
Silva Neo, A. V. B. d., Neo, G. d. S., Santos, M. D. F. d., Galvão Filho, K. J. A., and Freitas Junior, O. d. G. (2024). Revisão sobre a geração automática de questões na educação: Técnicas, conjuntos de dados e métricas de avaliação. In Anais do XIII Simpósio Brasileiro de Tecnologia da Informação (SBTI). SBTI.
Tan, B., Armoush, N., Mazzullo, E., Bulut, O., and Gierl, M. (2025). A review of automatic item generation techniques leveraging large language models. International Journal of Assessment Tools in Education, 12(2):317–340.
Wang, J., Xiao, R., and Tseng, Y.-J. (2025a). Generating ai literacy mcqs: A multi-agent llm approach. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 2, SIGCSETS 2025, pages 1651–1652, New York, NY, USA. Association for Computing Machinery.
Wang, Y., Yu, Z., Wang, Z., Yu, Z., and Wang, J. (2025b). Multi-examiner: A knowledge graph-driven system for generating comprehensive it questions with higher-order thinking. Applied Sciences, 15(10).
Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le, Q. V., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems (NeurIPS), pages 24824–24837. Curran Associates, Inc.
Yang, Y., Park, J., Zhou, Y., and Zhai, X. (2026). Developing a multi-agent system to generate next generation science assessments with evidence-centered design. AI4STEM Education Center, University of Georgia.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
Yao, Z., Parashar, A., Zhou, H., Jang, W. S., Ouyang, F., Yang, Z., and Yu, H. (2024). Mcqg-srefine: Multiple choice question generation and evaluation with iterative self-critique, correction, and comparison feedback. arXiv preprint arXiv:2410.01553.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J., and Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 46595–46623. Curran Associates, Inc.
Almond, R., Mislevy, R., and Steinberg, L. (2003). On the structure of educational assessments. Measurement: Interdisciplinary research and perspectives, 1(1):3–62.
Anderson, L. W. and Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy of educational objectives. Addison Wesley Longman, New York, complete edition edition.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., and Kaplan, J. (2022). Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
Behrens, J. T., Mislevy, R. J., DiCerbo, K. E., and Levy, R. (2010). An evidence centered design for learning and assessment in the digital world. CRESST Report 778, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Los Angeles, CA.
Brookhart, S. M. (2018). Appropriate criteria: Key to effective rubrics. Frontiers in Education, Volume 3 - 2018:22.
Caporal, A., Prado, M. R., Bini, I., and Boller, C. (2018). Padronização da correção de questões dissertativas para professores de saúde coletiva do curso de medicina em uma instituição de ensino superior do oeste do paraná. Revista Meta: Avaliação, 10(28):54–74.
Carvalho, F. S. and Soares, C. d. S. N. (2024). Avaliação por rubricas seguindo a taxonomia de bloom no diagnóstico de aprendizagem em algoritmos. In Anais do Simpósio Brasileiro de Informática na Educação.
Chan, K. W., Ali, F., Park, J., Sham, K. S. B., Tan, E. Y. T., Chong, F. W. C., Qian, K., and Sze, G. K. (2025). Automatic item generation in various stem subjects using large language model prompting. Computers and Education: Artificial Intelligence, 8:100344.
Cohn, C., Mohammed, N., Biswas, G., et al. (2025). Cotal: Human-in-the-loop prompt engineering for generalizable formative assessment scoring. arXiv preprint arXiv:2504.02323.
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., Bogart, C., Keylor, E., Kultur, C., Savelka, J., and Sakr, M. (2024). A comparative study of ai-generated (gpt-4) and human-crafted mcqs in programming education. In Proceedings of the 26th Australasian Computing Education Conference, ACE '24, pages 114–123, New York, NY, USA. Association for Computing Machinery.
Elkins, S., Kochmar, E., Cheung, J. C. K., and Serban, I. (2024). How teachers can use large language models and bloom's taxonomy to create educational quizzes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23084–23091.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv e-prints, page arXiv:2312.10997.
Gierl, M. J. and Haladyna, T. M., editors (2013). Automatic Item Generation: Theory and Practice. Routledge, New York, NY.
Gorgun, G. and Bulut, O. (2025). Instruction-tuned large-language models for quality control in automatic item generation: A feasibility study. Educational Measurement: Issues and Practice, 44(1):96–107.
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., and Fritz, M. (2023). Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, AISec '23, pages 79–90, New York, NY, USA. Association for Computing Machinery.
Grosmann, D., de Aguiar, R. K. B., da Silva, T. R., and Teixeira, M. M. (2025). Challenges, benefits, and perspectives on the use of virtual laboratories in computer networks teaching: A systematic mapping. In Sánchez, J., editor, Nuevas Ideas en Informática Educativa, volume 19, pages 87–97, Santiago de Chile.
Haladyna, T. M. (2004). Developing and Validating Multiple-Choice Test Items. Lawrence Erlbaum Associates, Inc., Mahwah, New Jersey, 3rd edition.
Haladyna, T. M. and Downing, S. M. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1):17–27.
Haladyna, T. M., Downing, S. M., and Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3):309–333.
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., and Liu, T. (2025). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2).
Jesus, G. R. d., Rêgo, R. M. d. L., and Souza, V. V. d. (2020). A questão da validade na avaliação educacional brasileira. Ensaio: Avaliação e Políticas Públicas em Educação.
Jia, R., Zhang, M., Liu, F., Jiang, B., Kuang, K., and Dai, Z. (2025). Eduagentqg: A multi-agent workflow framework for personalized question generation. arXiv preprint arXiv:2511.11635.
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. (2022). Language models (mostly) know what they know.
Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. (2020). Dense passage retrieval for open-domain question answering. arXiv e-prints, page arXiv:2004.04906.
Kubsch, M., Grimm, A., Neumann, K., Drachsler, H., and Rummel, N. (2024). Using evidence-centered design to develop an automated system for tracking students physics learning in a digital learning environment. In Zhai, X. and Krajcik, J., editors, Uses of Artificial Intelligence in STEM Education, chapter 11, pages 230–249. Oxford University Press.
Kurdi, G., Leo, J., Parsia, B., Sattler, U., and Al-Emari, S. (2020). A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30(1):121–204.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems (NeurIPS), pages 9459–9474. Curran Associates, Inc.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. (2023). Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems (NeurIPS), pages 46534–46594.
Maus, H., Tschisgale, P., Kieser, F., Petersen, S., and Wulff, P. (2025). Developing and evaluating a large language model-based automated feedback system grounded in evidence-centered design for supporting physics problem solving. IEEE Transactions on Learning Technologies.
Mislevy, R. J., Behrens, J. T., DiCerbo, K. E., and Levy, R. (2012). Design and discovery in educational assessment: Evidence-centered design, psychometrics, and educational data mining. Journal of Educational Data Mining, 4(1):11–48.
Mohammed, O. E. A. H., Srinivasamurthy, S. K., Bhat, R., Eltahir, Y. A. M. A., Elshamly, H. A. H., Mohammed, F., and Hamad, B. (2025). Postgraduate clinical residency: The impact of multiple-choice question quality on exam success rates. Advances in Medical Education and Practice, 16:1381–1397. PMID: 40786957.
Moore, S., Costello, E., Nguyen, H. A., and Stamper, J. (2024). An automatic question usability evaluation toolkit. In Olney, A. M., Chounta, I.-A., Liu, Z., Santos, O. C., and Bittencourt, I. I., editors, Artificial Intelligence in Education, pages 31–46, Cham. Springer Nature Switzerland.
Moore, S., Nguyen, H. A., Chen, T., and Stamper, J. (2023). Assessing the quality of multiple-choice questions using gpt-4 and rule-based methods. In Viberg, O., Jivet, I., Muñoz-Merino, P. J., Perifanou, M., and Papathoma, T., editors, Responsive and Sustainable Educational Futures, volume 14200, pages 229–245, Cham. Springer Nature Switzerland.
Popham, W. J. (1997). What's wrong and what's right-with rubrics. Educational leadership, 55(2):72–75.
Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. In Inui, K., Jiang, J., Ng, V., and Wan, X., editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
Schmucker, R. and Moore, S. (2025). The impact of item-writing flaws on difficulty and discrimination in item response theory. arXiv preprint arXiv:2503.10533.
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. arXiv e-prints, page arXiv:2303.11366.
Silva Neo, A. V. B. d., Neo, G. d. S., Santos, M. D. F. d., Galvão Filho, K. J. A., and Freitas Junior, O. d. G. (2024). Revisão sobre a geração automática de questões na educação: Técnicas, conjuntos de dados e métricas de avaliação. In Anais do XIII Simpósio Brasileiro de Tecnologia da Informação (SBTI). SBTI.
Tan, B., Armoush, N., Mazzullo, E., Bulut, O., and Gierl, M. (2025). A review of automatic item generation techniques leveraging large language models. International Journal of Assessment Tools in Education, 12(2):317–340.
Wang, J., Xiao, R., and Tseng, Y.-J. (2025a). Generating ai literacy mcqs: A multi-agent llm approach. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 2, SIGCSETS 2025, pages 1651–1652, New York, NY, USA. Association for Computing Machinery.
Wang, Y., Yu, Z., Wang, Z., Yu, Z., and Wang, J. (2025b). Multi-examiner: A knowledge graph-driven system for generating comprehensive it questions with higher-order thinking. Applied Sciences, 15(10).
Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le, Q. V., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems (NeurIPS), pages 24824–24837. Curran Associates, Inc.
Yang, Y., Park, J., Zhou, Y., and Zhai, X. (2026). Developing a multi-agent system to generate next generation science assessments with evidence-centered design. AI4STEM Education Center, University of Georgia.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
Yao, Z., Parashar, A., Zhou, H., Jang, W. S., Ouyang, F., Yang, Z., and Yu, H. (2024). Mcqg-srefine: Multiple choice question generation and evaluation with iterative self-critique, correction, and comparison feedback. arXiv preprint arXiv:2410.01553.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J., and Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems (NeurIPS), volume 36, pages 46595–46623. Curran Associates, Inc.
Publicado
05/10/2026
Como Citar
GROSMANN, Diego; SILVA, Thiago Reis; OLIVEIRA, Alana; TEIXEIRA, Mario Meireles.
EnsinaAI: Geração de Questões com Qualidade Pedagógica Auditada por Catálogos Declarativos. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 37. , 2026, Goiânia/GO.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 2644-2656.
DOI: https://doi.org/10.5753/sbie.2026.27699.
