Detecção Automática de Clímax em Produções de Textos Narrativos
Resumo
A correção automática de redações é um problema que vem sendo bastante explorado nos últimos anos. Um dos aspectos mais desafiadores nessa tarefa é avaliação do nível de domínio do aluno quanto aos mais variados tipos de estruturas textuais. A estrutura narrativa é um caso especialmente complexo devido ao seu caráter extremamente subjetivo. Trabalhos anteriores na área de correção textual, não abordaram o problema de automatizar a avaliação do nível de competência do aluno na escrita de narrativas. Este trabalho investiga o uso de algoritmos de aprendizagem de máquina para a detecção de clímax em redações em Português como um passo inicial na resolução do problema de correção automática de textos narrativos. Três algoritmos de classificação tradicionais, o support vector machine, floresta aleatória e descida de gradiente estocástica, foram aplicados em um conjunto de dados anotado traduzido para o Português. Os algoritmos foram avaliados em termos de precisão, revocação e pontuação F1, sendo a floresta aleatória o algoritmo de melhor desempenho. Além disso, foi realizado uma análise dos atributos envolvidos, e os experimentos mostraram que os melhores resultados são obtidos ao combinar-se atributos tanto do Coh-Metrix quanto do LIWC.
Palavras-chave:
Aprendizado de Máquina, PLN, Classificação Textual
Referências
Al-Anzi, F. S. (2022a). An effective hybrid stochastic gradient descent arabic sentiment analysis with partial-order microwords and piecewise differentiation. In 2022 9th International Conference on Electrical and Electronics Engineering (ICEEE), pages 408–411. IEEE.
Al-Anzi, F. S. (2022b). An effective hybrid stochastic gradient descent for classification of short text communication in e-learning environments. In 2022 8th International Conference on Control, Decision and Information Technologies (CoDIT), volume 1, pages 1096–1101. IEEE.
Balage Filho, P., Pardo, T. A. S., and Aluísio, S. (2013). An evaluation of the brazilian portuguese liwc dictionary for sentiment analysis. In Proceedings of the 9th Brazilian Symposium in Information and Human Language Technology.
Burstein, J., Leacock, C., and Swartz, R. (2001). Automated evaluation of essays and short answers.
Cavalcanti, A. P., Diego, A., Mello, R. F., Mangaroska, K., Nascimento, A., Freitas, F., and Gašević, D. (2020). How good is my feedback? a content analysis of written feedback. In Proceedings of the tenth international conference on learning analytics & knowledge, pages 428–437.Brasileiro de Informática na Educação, pages 179–186, Porto Alegre, RS, Brasil. SBC.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.
Costa, L., Oliveira, E., and Junior, A. C. (2020). Corretor automático de redações em língua portuguesa: um mapeamento sistemático de literatura. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 1403–1412, Porto Alegre, RS, Brasil. SBC.
Fernández-Delgado, M., Cernadas, E., Barro, S., and Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems? The journal of machine learning research, 15(1):3133–3181.
Ferreira-Mello, R., André, M., Pinheiro, A., Costa, E., and Romero, C. (2019). Text mining in education. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(6):e1332.
Foltz, P. W., Laham, D., and Landauer, T. K. (1999). The intelligent essay assessor: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1(2):939–944.
Fonseca, E., Medeiros, I., Kamikawachi, D., and Bokan, A. (2018). Automatically grading brazilian student essays. In International Conference on Computational Processing of the Portuguese Language, pages 170–179. Springer
Francis, M. and Booth, R. J. (1993). Linguistic inquiry and word count. Southern Methodist University: Dallas, TX, USA.
Freitas, E., Falcão, T. P., and Mello, R. F. (2020). Desmistificando a adoção de learning analytics: um guia conciso sobre ferramentas e instrumentos. In Jornada de Atualização em Informática na Educação, pages 73-99. Sociedade Brasileira de Computação.
Hastie, T., Tibshirani, R., Friedman, J. H., and Friedman, J. H. (2009). The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer.
Hossin, M. and Sulaiman, M. N. (2015). A review on evaluation metrics for data classification evaluations. International journal of data mining & knowledge management process, 5(2):1.
Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., and Siemens, G. (2016). Towards automated content analysis of discussion transcripts: A cognitive presence case. In Proceedings of the sixth international conference on learning analytics & knowledge, pages 15–24.
Labov, W. and Waletzky, J. (1967). Narrative analysis. essays on the verbal and visual arts, ed. by june helm, 12-44.
Labov, W. and Waletzky, J. (1997). Narrative analysis: Oral versions of personal experience.
McHugh, M. L. (2013). The chi-square test of independence. Biochemia medica, 23(2):143–149.
McNamara, D. S., Graesser, A. C., McCarthy, P. M., and Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press.
Ouyang, J. and McKeown, K. (2014). Towards automatic detection of narrative structure. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4624–4631, Reykjavik, Iceland. European Language Resources Association (ELRA).
Passero, G., Ferreira, R., and Dazzi, R. L. S. (2019). Off-topic essay detection: A comparative study on the portuguese language. Revista Brasileira de Informática na Educação, 27(03):177–190.
Rajkumar, A. and Agarwal, S. (2012). A differentially private stochastic gradient descent algorithm for multiparty classification. In Artificial Intelligence and Statistics, pages 933–941. PMLR.
Ramesh, D. and Sanampudi, S. K. (2021). An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review, pages 1–33.
Rudner, L. M. and Gagne, P. (2000). An overview of three approaches to scoring written essays by computer. Practical Assessment, Research, and Evaluation, 7(1):26.
Scarton, C. E. and Aluísio, S. M. (2010). Análise da inteligibilidade de textos via ferramentas de processamento de língua natural: adaptando as métricas do coh-metrix para o português. Linguamática, 2(1):45–61.
Sousa, E. B. d., Alexandre, B., Ferreira Mello, R., Pontual Falcão, T., Vesin, B., and Gašević, D. (2021). Applications of learning analytics in high schools: a systematic literature review. Frontiers in Artificial Intelligence, 4:737891.
Van Wissen, L. and Boot, P. (2017). An electronic translation of the liwc dictionary into dutch. In Electronic lexicography in the 21st century: Proceedings of eLex 2017 conference, pages 703–715. Lexical Computing.
Vijaya Shetty, S., Guruvyas, K., Patil, P. P., and Acharya, J. J. (2022). Essay scoring systems using ai and feature extraction: A review. In Proceedings of Third International Conference on Communication, Computing and Electronics Systems, pages 45–57. Springer.
Wang, Z., Liu, J., and Dong, R. (2018). Intelligent auto-grading system. In 2018 5th IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS), pages 430–435. IEEE.
You, K. and Goldwasser, D. (2020). "where is this relationship going?": Understanding relationship trajectories in narrative text. arXiv preprint arXiv:2010.15313.
Zhang, T. (2004). Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the twenty-first international conference on Machine learning, page 116.
Al-Anzi, F. S. (2022b). An effective hybrid stochastic gradient descent for classification of short text communication in e-learning environments. In 2022 8th International Conference on Control, Decision and Information Technologies (CoDIT), volume 1, pages 1096–1101. IEEE.
Balage Filho, P., Pardo, T. A. S., and Aluísio, S. (2013). An evaluation of the brazilian portuguese liwc dictionary for sentiment analysis. In Proceedings of the 9th Brazilian Symposium in Information and Human Language Technology.
Burstein, J., Leacock, C., and Swartz, R. (2001). Automated evaluation of essays and short answers.
Cavalcanti, A. P., Diego, A., Mello, R. F., Mangaroska, K., Nascimento, A., Freitas, F., and Gašević, D. (2020). How good is my feedback? a content analysis of written feedback. In Proceedings of the tenth international conference on learning analytics & knowledge, pages 428–437.Brasileiro de Informática na Educação, pages 179–186, Porto Alegre, RS, Brasil. SBC.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.
Costa, L., Oliveira, E., and Junior, A. C. (2020). Corretor automático de redações em língua portuguesa: um mapeamento sistemático de literatura. In Anais do XXXI Simpósio Brasileiro de Informática na Educação, pages 1403–1412, Porto Alegre, RS, Brasil. SBC.
Fernández-Delgado, M., Cernadas, E., Barro, S., and Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems? The journal of machine learning research, 15(1):3133–3181.
Ferreira-Mello, R., André, M., Pinheiro, A., Costa, E., and Romero, C. (2019). Text mining in education. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(6):e1332.
Foltz, P. W., Laham, D., and Landauer, T. K. (1999). The intelligent essay assessor: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1(2):939–944.
Fonseca, E., Medeiros, I., Kamikawachi, D., and Bokan, A. (2018). Automatically grading brazilian student essays. In International Conference on Computational Processing of the Portuguese Language, pages 170–179. Springer
Francis, M. and Booth, R. J. (1993). Linguistic inquiry and word count. Southern Methodist University: Dallas, TX, USA.
Freitas, E., Falcão, T. P., and Mello, R. F. (2020). Desmistificando a adoção de learning analytics: um guia conciso sobre ferramentas e instrumentos. In Jornada de Atualização em Informática na Educação, pages 73-99. Sociedade Brasileira de Computação.
Hastie, T., Tibshirani, R., Friedman, J. H., and Friedman, J. H. (2009). The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer.
Hossin, M. and Sulaiman, M. N. (2015). A review on evaluation metrics for data classification evaluations. International journal of data mining & knowledge management process, 5(2):1.
Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., and Siemens, G. (2016). Towards automated content analysis of discussion transcripts: A cognitive presence case. In Proceedings of the sixth international conference on learning analytics & knowledge, pages 15–24.
Labov, W. and Waletzky, J. (1967). Narrative analysis. essays on the verbal and visual arts, ed. by june helm, 12-44.
Labov, W. and Waletzky, J. (1997). Narrative analysis: Oral versions of personal experience.
McHugh, M. L. (2013). The chi-square test of independence. Biochemia medica, 23(2):143–149.
McNamara, D. S., Graesser, A. C., McCarthy, P. M., and Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press.
Ouyang, J. and McKeown, K. (2014). Towards automatic detection of narrative structure. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4624–4631, Reykjavik, Iceland. European Language Resources Association (ELRA).
Passero, G., Ferreira, R., and Dazzi, R. L. S. (2019). Off-topic essay detection: A comparative study on the portuguese language. Revista Brasileira de Informática na Educação, 27(03):177–190.
Rajkumar, A. and Agarwal, S. (2012). A differentially private stochastic gradient descent algorithm for multiparty classification. In Artificial Intelligence and Statistics, pages 933–941. PMLR.
Ramesh, D. and Sanampudi, S. K. (2021). An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review, pages 1–33.
Rudner, L. M. and Gagne, P. (2000). An overview of three approaches to scoring written essays by computer. Practical Assessment, Research, and Evaluation, 7(1):26.
Scarton, C. E. and Aluísio, S. M. (2010). Análise da inteligibilidade de textos via ferramentas de processamento de língua natural: adaptando as métricas do coh-metrix para o português. Linguamática, 2(1):45–61.
Sousa, E. B. d., Alexandre, B., Ferreira Mello, R., Pontual Falcão, T., Vesin, B., and Gašević, D. (2021). Applications of learning analytics in high schools: a systematic literature review. Frontiers in Artificial Intelligence, 4:737891.
Van Wissen, L. and Boot, P. (2017). An electronic translation of the liwc dictionary into dutch. In Electronic lexicography in the 21st century: Proceedings of eLex 2017 conference, pages 703–715. Lexical Computing.
Vijaya Shetty, S., Guruvyas, K., Patil, P. P., and Acharya, J. J. (2022). Essay scoring systems using ai and feature extraction: A review. In Proceedings of Third International Conference on Communication, Computing and Electronics Systems, pages 45–57. Springer.
Wang, Z., Liu, J., and Dong, R. (2018). Intelligent auto-grading system. In 2018 5th IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS), pages 430–435. IEEE.
You, K. and Goldwasser, D. (2020). "where is this relationship going?": Understanding relationship trajectories in narrative text. arXiv preprint arXiv:2010.15313.
Zhang, T. (2004). Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the twenty-first international conference on Machine learning, page 116.
Publicado
16/11/2022
Como Citar
BATISTA, Hyan H. N.; BARBOSA, Gabriel A.; MIRANDA, Péricles; SANTOS, Jário; ISOTANI, Seiji; CORDEIRO, Thiago; BITTENCOURT, Ig Ibert; FERREIRA MELLO, Rafael.
Detecção Automática de Clímax em Produções de Textos Narrativos. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO (SBIE), 33. , 2022, Manaus.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 932-943.
DOI: https://doi.org/10.5753/sbie.2022.224770.
