Automatic Generation of Large-Scale Assessment Questions
Resumo
Assessment is the means used to diagnose learning difficulties. Assessment through questionnaires is one of the ways used to measure the level of content learning by students. Manually elaborating questions is time-consuming and makes the teacher uncomfortable thinking about different questions. Using an automatic question generator allows the teacher to access several questions and apply automatic correction, disseminating assessment results more quickly. If we analyze Massive Open Online Courses (MOOCs), it becomes humanly impossible to correct the large volume of questions manually. We present an institutional tool that recognizes chemical entities from free texts inserted by the teacher and automatically generates questions and their respective answers. We recognized the chemical entities through Local Grammar (LG), and then we elaborated free-text and fill-in-the-blank questions through triplets with spaCy in Python language. To validate the tool, we used two recognized books in Organic Chemistry. Chemistry specialists validated and indicated the level of difficulty of the questions. We generated 64 questions, 33 of which were free-text questions, and 31 were fill-in-the-blank questions. The average of the experts’ assessment indicated that 78,77% (average) and 87,5% (median) of the questions have quality and that most questions have an intermediate difficulty level. This research suggests as main contributions the recognition of named entities in the chemical area and the automatic generation of questions in Portuguese. These tasks can facilitate the human being's job, reducing the time for generating and correcting questions and reducing institutional costs with such activities.
Palavras-chave:
Chemistry Questions, Named Entity Recognition, Natural Language Processing, Artificial Intelligence
Referências
Mahmoud J Abu Ghali, Abdullah Abu Ayyad, Samy S Abu-Naser, and Mousa Abu Laban. 2018. An intelligent tutoring system for teaching English grammar. International Journal of Academic Engineering Research 2 (2018), 6.
Naveed Afzal and Ruslan Mitkov. 2014. Automatic generation of multiple choice questions using dependency-based semantic relations. Soft Computing 18(2014), 13. Issue 7. https://doi.org/10.1007/s00500-013-1141-4
Marcus Amorim, Jefferson Simões, Felipe Assis, João Pinheiro, Daniel Menasch, Claudia Motta, and Ageu Pacheco. 2019. Aumentando a Interatividade no Ensino a Distância via Geração Automática de Questões: Desafios, Soluções via Aprendizado por Máquina e Um Estudo de Caso no CEDERJ. In Anais do XXVII Workshop sobre Educação em Computação (Belém). SBC, Porto Alegre, RS, Brasil, 188–202. https://doi.org/10.5753/wei.2019.6629
Vanessa Castillo-Ramoran. 2008. Web Tool for Teachers: Information Systems on Categorized Teacher Materials and Online Examination. In Innovative Techniques in Instruction Technology, E-learning, E-assessment, and Education, Magued Iskander (Ed.). Springer Netherlands, Dordrecht, 570–575.
Sônia Maria Oliveira Costa and Jane Eire Silva Alencar Menezes. 2015. Química Orgânica I(2 ed.). Editora EdUECE, Fortaleza. https://educapes.capes.gov.br/handle/capes/431846
Juliana Pirovani e Marcos Spalenza e Elias Oliveira. 2017. Geração Automática de Questões a Partir do Reconhecimento de Entidades Nomeadas em Textos Didáticos. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação - SBIE) 28, 1 (2017), 1147. https://doi.org/10.5753/cbie.sbie.2017.1147
Ibrahim Eldesoky Fattoh, Amal Elsayed Aboutabl, and Mohamed Hassan Haggag. 2014. Article: Semantic Attributes Model for Automatic Generation of Multiple Choice Questions. International Journal of Computer Applications 103, 1 (October 2014), 18–24. Full text available.
Ricardo Feltre. 2008. Química - Química Orgânica(7 ed.). Vol. 3. Editora Moderna, São Paulo.
Bruno Gonçalves and Vitor Gonçalves. 2014. MOOC: uma estratégia de captação de alunos?Revista Multimédia de Investigação em Educação 1(2014), 15. http://hdl.handle.net/10198/11499
Carlos González-Sancho and Stéphan Vincent-Lancrin. 2016. Transforming education by using a new generation of information systems. Policy Futures in Education 14, 6 (2016), 741–758. https://doi.org/10.1177/1478210316649287
Takuya Goto, Tomoko Kojiri, Toyohide Watanabe, Tomoharu Iwata, and Takeshi Yamada. 2010. Automatic generation system of multiple-choice cloze questions and its evaluation. Knowledge Management & E-Learning: An International Journal 2, 3(2010), 210–224. https://doi.org/10.34105/j.kmel.2010.02.016
Jennifer Hill and Rahul Simha. 2016. Automatic Generation of Context-Based Fill-in-the-Blank Exercises Using Co-occurrence Likelihoods and Google n-grams. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, San Diego, CA, 23–30. https://doi.org/10.18653/v1/W16-0503
Anselma Regina Levorato, Arthur Auwerter, Belmayr Knopki Nery, Elisabete Soares Cebulski, Jussara Turin Politano, Maria Bernadete Pimenta Buzatto, Miriam Goretti Stingelin Nepomoceno, and Zecliz Stadler. 2006. Química – Ensino Médio(2 ed.). SEED-PR, Curitiba. 248 pages.
Shlok Kumar Mishra, Pranav Goel, Abhishek Sharma, Abhyuday Jagannatha, David Jacobs, and Hal Daumé III au2. 2020. Towards Automatic Generation of Questions from Long Answers. arxiv:2004.05109
Cristina Mota and Diana Santos. 2008. Desafios na avaliação conjunta do reconhecimento de entidades mencionadas: O Segundo HAREM. Linguateca, Porto.
Katya Luciane de Oliveira and Acácia Aparecida Angeli dos Santos. 2005. Avaliação da aprendizagem na univesidade. Psicologia Escolar e Educacional 9 (06 2005), 37–46. https://doi.org/10.1590/S1413-85572005000100004
Sumeet Pannu, Aishwarya Krishna, Shiwani Kumari, Rakesh Patra, and Sujan Kumar Saha. 2018. Automatic Generation of Fill-in-the-Blank Questions From History Books for School-Level Evaluation. In Progress in Computing, Analytics and Networking, Prasant Kumar Pattnaik, Siddharth Swarup Rautaray, Himansu Das, and Janmenjoy Nayak (Eds.). Springer Singapore, Singapore, 461–469.
Philippe Perrenoud and Monica Gather Thurler. 2009. As competências para ensinar no século XXI: a formação dos professores e o desafio da avaliação. Artmed Editora, Porto Alegre.
Juliana Pirovani and Elias Oliveira. 2021. Studying the Adaptation of Portuguese NER for Different Textual Genres. The Journal of Supercomputing 77, 3801 (2021), 16. https://doi.org/10.1007/s11227-021-03801-9
Ingrid Rewitzky. 2001. Towards Automated Testing. https://hdl.handle.net/2134/1829
Maksim Tkachenko and Andrey Simanovsky. 2012. Named entity recognition: Exploring features. In Proceedings of KONVENS 2012, Jeremy Jancsary (Ed.). ÖGAI, Vienna, 118–127. http://www.oegai.at/konvens2012/proceedings/17_tkachenko12o/ Main track: oral presentations.
Maria Cândida Trigo. 2011. Avaliação Educacional. In Avaliação Formativa: revendo decisões e ações educativas. Fundação Cecierj/Consórcio Cederj, Rio de Janeiro, 56–60.
Naveed Afzal and Ruslan Mitkov. 2014. Automatic generation of multiple choice questions using dependency-based semantic relations. Soft Computing 18(2014), 13. Issue 7. https://doi.org/10.1007/s00500-013-1141-4
Marcus Amorim, Jefferson Simões, Felipe Assis, João Pinheiro, Daniel Menasch, Claudia Motta, and Ageu Pacheco. 2019. Aumentando a Interatividade no Ensino a Distância via Geração Automática de Questões: Desafios, Soluções via Aprendizado por Máquina e Um Estudo de Caso no CEDERJ. In Anais do XXVII Workshop sobre Educação em Computação (Belém). SBC, Porto Alegre, RS, Brasil, 188–202. https://doi.org/10.5753/wei.2019.6629
Vanessa Castillo-Ramoran. 2008. Web Tool for Teachers: Information Systems on Categorized Teacher Materials and Online Examination. In Innovative Techniques in Instruction Technology, E-learning, E-assessment, and Education, Magued Iskander (Ed.). Springer Netherlands, Dordrecht, 570–575.
Sônia Maria Oliveira Costa and Jane Eire Silva Alencar Menezes. 2015. Química Orgânica I(2 ed.). Editora EdUECE, Fortaleza. https://educapes.capes.gov.br/handle/capes/431846
Juliana Pirovani e Marcos Spalenza e Elias Oliveira. 2017. Geração Automática de Questões a Partir do Reconhecimento de Entidades Nomeadas em Textos Didáticos. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação - SBIE) 28, 1 (2017), 1147. https://doi.org/10.5753/cbie.sbie.2017.1147
Ibrahim Eldesoky Fattoh, Amal Elsayed Aboutabl, and Mohamed Hassan Haggag. 2014. Article: Semantic Attributes Model for Automatic Generation of Multiple Choice Questions. International Journal of Computer Applications 103, 1 (October 2014), 18–24. Full text available.
Ricardo Feltre. 2008. Química - Química Orgânica(7 ed.). Vol. 3. Editora Moderna, São Paulo.
Bruno Gonçalves and Vitor Gonçalves. 2014. MOOC: uma estratégia de captação de alunos?Revista Multimédia de Investigação em Educação 1(2014), 15. http://hdl.handle.net/10198/11499
Carlos González-Sancho and Stéphan Vincent-Lancrin. 2016. Transforming education by using a new generation of information systems. Policy Futures in Education 14, 6 (2016), 741–758. https://doi.org/10.1177/1478210316649287
Takuya Goto, Tomoko Kojiri, Toyohide Watanabe, Tomoharu Iwata, and Takeshi Yamada. 2010. Automatic generation system of multiple-choice cloze questions and its evaluation. Knowledge Management & E-Learning: An International Journal 2, 3(2010), 210–224. https://doi.org/10.34105/j.kmel.2010.02.016
Jennifer Hill and Rahul Simha. 2016. Automatic Generation of Context-Based Fill-in-the-Blank Exercises Using Co-occurrence Likelihoods and Google n-grams. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, San Diego, CA, 23–30. https://doi.org/10.18653/v1/W16-0503
Anselma Regina Levorato, Arthur Auwerter, Belmayr Knopki Nery, Elisabete Soares Cebulski, Jussara Turin Politano, Maria Bernadete Pimenta Buzatto, Miriam Goretti Stingelin Nepomoceno, and Zecliz Stadler. 2006. Química – Ensino Médio(2 ed.). SEED-PR, Curitiba. 248 pages.
Shlok Kumar Mishra, Pranav Goel, Abhishek Sharma, Abhyuday Jagannatha, David Jacobs, and Hal Daumé III au2. 2020. Towards Automatic Generation of Questions from Long Answers. arxiv:2004.05109
Cristina Mota and Diana Santos. 2008. Desafios na avaliação conjunta do reconhecimento de entidades mencionadas: O Segundo HAREM. Linguateca, Porto.
Katya Luciane de Oliveira and Acácia Aparecida Angeli dos Santos. 2005. Avaliação da aprendizagem na univesidade. Psicologia Escolar e Educacional 9 (06 2005), 37–46. https://doi.org/10.1590/S1413-85572005000100004
Sumeet Pannu, Aishwarya Krishna, Shiwani Kumari, Rakesh Patra, and Sujan Kumar Saha. 2018. Automatic Generation of Fill-in-the-Blank Questions From History Books for School-Level Evaluation. In Progress in Computing, Analytics and Networking, Prasant Kumar Pattnaik, Siddharth Swarup Rautaray, Himansu Das, and Janmenjoy Nayak (Eds.). Springer Singapore, Singapore, 461–469.
Philippe Perrenoud and Monica Gather Thurler. 2009. As competências para ensinar no século XXI: a formação dos professores e o desafio da avaliação. Artmed Editora, Porto Alegre.
Juliana Pirovani and Elias Oliveira. 2021. Studying the Adaptation of Portuguese NER for Different Textual Genres. The Journal of Supercomputing 77, 3801 (2021), 16. https://doi.org/10.1007/s11227-021-03801-9
Ingrid Rewitzky. 2001. Towards Automated Testing. https://hdl.handle.net/2134/1829
Maksim Tkachenko and Andrey Simanovsky. 2012. Named entity recognition: Exploring features. In Proceedings of KONVENS 2012, Jeremy Jancsary (Ed.). ÖGAI, Vienna, 118–127. http://www.oegai.at/konvens2012/proceedings/17_tkachenko12o/ Main track: oral presentations.
Maria Cândida Trigo. 2011. Avaliação Educacional. In Avaliação Formativa: revendo decisões e ações educativas. Fundação Cecierj/Consórcio Cederj, Rio de Janeiro, 56–60.
Publicado
16/05/2022
Como Citar
IZO, Flavio; LEÃO, Jhonatan; PIROVANI, Juliana Pinheiro Campos; OLIVEIRA, Elias.
Automatic Generation of Large-Scale Assessment Questions. In: SIMPÓSIO BRASILEIRO DE SISTEMAS DE INFORMAÇÃO (SBSI), 18. , 2022, Curitiba.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
