Abordagens, Práticas e Desafios da Avaliação Automática de Exercícios de Programação
Resumo
A avaliação automática de programação surgiu como um importante método para auxiliar professores na correção de exercícios e possibilitar feedbacks imediatos a estudantes, principalmente em turmas numerosas. As soluções propostas têm avaliado itens como a execução correta de um programa, a codificação, o estilo de programação, as métricas de software, o plágio e até aspectos psicológicos. Mas, embora os sistemas de avaliação automática de programação tenham evoluído, tais sistemas ainda são limitados em avaliar se de fato objetivos educacionais foram alcançados. Dessa forma, entre tantos desafios, o principal desafio da avaliação automática de programação é avaliar o que de fato um professor de programação quer avaliar. Com os objetivos de discutir e propor soluções para esse desafio, este trabalho apresenta um sumário do estado da arte da avaliação automática de programação e aponta caminhos para tornar essa avaliação funcional nos próximos dez anos.
Referências
Ala-Mutka, K. M. (2005). A survey of automated assessment approaches for programming assignments. Computer Science Education, 15(2):83–102.
Benford, S. D., Burke, E. K., Foxley, E., and Higgins, C. A. (1995). The Ceilidh system for the automatic grading of students on programming courses. In Proceedings of the 33rd annual on Southeast regional conference, ACM-SE 33, pages 176–182, New York, NY, USA. ACM.
Blumenstein, M. M., Green, S., Nguyen, A. T., and Muthukkumarasamy, V. (2004). Game: A generic automated marking environment for programming assessment.
Campos, C. and Ferreira, C. (2004). Boca: um sistema de apoio para competições de programação. In XII Workshop de Educação em Computação (WEI) - SBC 2004), Salvador, BA.
C.M. Tang, Y. Y. and Poon, C. (2009). An approach towards automatic testing of student programs using token patterns. In Proceedings of the 17th International Conference on Computers in Education, pages 118–190, Hong Kong.
Curtis, B., Sheppard, S. B.,Milliman, P., Borst,M. A., and Love, T. (1979). Measuring the psychological complexity of software maintenance tasks with the halstead and mccabe metrics. IEEE Trans. Softw. Eng., 5(2):96–104.
De Souza, D., Maldonado, J., and Barbosa, E. (2011). Progtest: An environment for the submission and evaluation of programming assignments based on testing activities. In Software Engineering Education and Training (CSEE T), 2011 24th IEEE-CS Conference on, pages 1 –10.
Dias, P. (2001). Avaliação automática de exercícios em sql. Master’s thesis, Faculdade de Engenharia da Universidade do Porto, Porto, Portugal.
Ebrahimi, A. (1994). Novice programmer errors: language constructs and plan composition. International Journal of Human-Computer Studies, 41(4):457 – 480.
Franca, A., Soares, J., Gomes, D., and G.C.Barroso (2011). Um sistema orientado a serviços para suporte a atividades de laboratório em disciplinas de técnicas de programacão com integracão ao ambiente moodle. RENOTE - Revista Novas Tecnologias na Educação, 9(1).
Gerdes, A., Jeuring, J. T., and Heeren, B. J. (2010). Using strategies for assessment of programming exercises. In Proceedings of the 41st ACM technical symposium on Computer science education, SIGCSE ’10, pages 441–445, New York, NY, USA. ACM.
Hollingsworth, J. (1960). Automatic graders for programming classes. Commun. ACM, 3(10):528–529.
Hung, S.-l., Kwok, L.-f., and Chung, A. (1993). New metrics for automated programming assessment. In Proceedings of the IFIP WG3.4/SEARCC (SRIG on Education and Training) Working Conference on Software Engineering Education, pages 233–243, Amsterdam, The Netherlands, The Netherlands. North-Holland Publishing Co.
Ihantola, P., Ahoniemi, T., Karavirta, V., and Seppälä, O. (2010). Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, Koli Calling ’10, pages 86–93, New York, NY, USA. ACM.
Jackson, D. and Usher, M. (1997). Grading student programs using assyst. SIGCSE Bull., 29(1):335–339.
Malmi, L., Korhonen, A., and Saikkonen, R. (2002). Experiences in automatic assessment on mass courses and issues for designing virtual courses. In Proceedings of the 7th annual conference on Innovation and technology in computer science education, ITiCSE ’02, pages 55–59, New York, NY, USA. ACM.
Moreira, M. P. and Favero, E. L. (2009). Um ambiente para ensino de programação com feedback automático de exercícios. In XVII Workshop Sobre Educação em Computação (WEI) - CSBC 2009.
Moussiades, L. and Vakali, A. (2005). Pdetect: A clustering approach for detecting plagiarism in source code datasets. The computer journal, 48(6):651–661.
Naude, K. A., Greyling, J. H., and Vogts, D. (2010). Marking student programs using graph similarity. Computers & Education, 54(2):545 – 561.
Nunes, I. and Lisboa,M. (2004). Testador automático emétodo de avaliacão de programas em java. In XVI Salão de Iniciacão Científica e XIII Feira de Iniciacão Científica, Porto Alegre, RS.
Orcero, D. S. (2000). The code analyser LClint. Linux J., 2000(73).
Pieterse, V. (2013). Automated assessment of programming assignments. In Proceedings of the 3rd Computer Science Education Research Conference on Computer Science Education Research, CSERC ’13, pages 4:45–4:56, Open Univ., Heerlen, The Netherlands, The Netherlands. Open Universiteit, Heerlen.
Rahman, K. A., Ahmad, S., Nordin, M. J., and Maklumat, F. T. D. S. (2008). The Design of an Automated C Programming Assessment Using Pseudo-code Comparison Technique.
Rees, M. J. (1982). Automatic assessment aids for pascal programs. SIGPLAN Not., 17:33–42.
Romli, R., Sulaiman, S., and Zamli, K. (2010). Automatic programming assessment and test data generation a review on its approaches. In Information Technology (ITSim), 2010 International Symposium in, volume 3, pages 1186 –1192.
Saikkonen, R., Malmi, L., and Korhonen, A. (2001). Fully automatic assessment of programming exercises. SIGCSE Bull., 33:133–136.
Truong, N., Roe, P., and Bancroft, P. (2004). Static analysis of students’ java programs. In Proceedings of the Sixth Australasian Conference on Computing Education - Volume 30, ACE ’04, pages 317–325, Darlinghurst, Australia, Australia. Australian Computer Society, Inc.
Vujosevic-Janicic, M., Nikolic, M., Tosic, D., and Kuncak, V. (2013). Software verification and graph similarity for automated evaluation of students assignments. Information and Software Technology, 55(6):1004 – 1016.
Wang, T., Su, X., Ma, P., Wang, Y., and Wang, K. (2011). Ability-training-oriented automated assessment in introductory programming course. Comput. Educ., 56(1):220–226.
Wu,W., Li, G., Sun, Y.,Wang, J., and Lai, T. (2007). AnalyseC: A Framework for Assessing Students’ Programs at Structural and Semantic Level. In Control and Automation, 2007. ICCA 2007. IEEE International Conference on, pages 742 –747.
Xu, S. and Chee, Y. S. (2003). Transformation-based diagnosis of student programs for programming tutoring systems. IEEE Transactions on Software Engineering, 29:360–384.
Zin, A. and Foxley, E. (1994). Analyse:an automatic program assessment system. Malaysian Journal of Computer Science, 7:123–142.
Benford, S. D., Burke, E. K., Foxley, E., and Higgins, C. A. (1995). The Ceilidh system for the automatic grading of students on programming courses. In Proceedings of the 33rd annual on Southeast regional conference, ACM-SE 33, pages 176–182, New York, NY, USA. ACM.
Blumenstein, M. M., Green, S., Nguyen, A. T., and Muthukkumarasamy, V. (2004). Game: A generic automated marking environment for programming assessment.
Campos, C. and Ferreira, C. (2004). Boca: um sistema de apoio para competições de programação. In XII Workshop de Educação em Computação (WEI) - SBC 2004), Salvador, BA.
C.M. Tang, Y. Y. and Poon, C. (2009). An approach towards automatic testing of student programs using token patterns. In Proceedings of the 17th International Conference on Computers in Education, pages 118–190, Hong Kong.
Curtis, B., Sheppard, S. B.,Milliman, P., Borst,M. A., and Love, T. (1979). Measuring the psychological complexity of software maintenance tasks with the halstead and mccabe metrics. IEEE Trans. Softw. Eng., 5(2):96–104.
De Souza, D., Maldonado, J., and Barbosa, E. (2011). Progtest: An environment for the submission and evaluation of programming assignments based on testing activities. In Software Engineering Education and Training (CSEE T), 2011 24th IEEE-CS Conference on, pages 1 –10.
Dias, P. (2001). Avaliação automática de exercícios em sql. Master’s thesis, Faculdade de Engenharia da Universidade do Porto, Porto, Portugal.
Ebrahimi, A. (1994). Novice programmer errors: language constructs and plan composition. International Journal of Human-Computer Studies, 41(4):457 – 480.
Franca, A., Soares, J., Gomes, D., and G.C.Barroso (2011). Um sistema orientado a serviços para suporte a atividades de laboratório em disciplinas de técnicas de programacão com integracão ao ambiente moodle. RENOTE - Revista Novas Tecnologias na Educação, 9(1).
Gerdes, A., Jeuring, J. T., and Heeren, B. J. (2010). Using strategies for assessment of programming exercises. In Proceedings of the 41st ACM technical symposium on Computer science education, SIGCSE ’10, pages 441–445, New York, NY, USA. ACM.
Hollingsworth, J. (1960). Automatic graders for programming classes. Commun. ACM, 3(10):528–529.
Hung, S.-l., Kwok, L.-f., and Chung, A. (1993). New metrics for automated programming assessment. In Proceedings of the IFIP WG3.4/SEARCC (SRIG on Education and Training) Working Conference on Software Engineering Education, pages 233–243, Amsterdam, The Netherlands, The Netherlands. North-Holland Publishing Co.
Ihantola, P., Ahoniemi, T., Karavirta, V., and Seppälä, O. (2010). Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, Koli Calling ’10, pages 86–93, New York, NY, USA. ACM.
Jackson, D. and Usher, M. (1997). Grading student programs using assyst. SIGCSE Bull., 29(1):335–339.
Malmi, L., Korhonen, A., and Saikkonen, R. (2002). Experiences in automatic assessment on mass courses and issues for designing virtual courses. In Proceedings of the 7th annual conference on Innovation and technology in computer science education, ITiCSE ’02, pages 55–59, New York, NY, USA. ACM.
Moreira, M. P. and Favero, E. L. (2009). Um ambiente para ensino de programação com feedback automático de exercícios. In XVII Workshop Sobre Educação em Computação (WEI) - CSBC 2009.
Moussiades, L. and Vakali, A. (2005). Pdetect: A clustering approach for detecting plagiarism in source code datasets. The computer journal, 48(6):651–661.
Naude, K. A., Greyling, J. H., and Vogts, D. (2010). Marking student programs using graph similarity. Computers & Education, 54(2):545 – 561.
Nunes, I. and Lisboa,M. (2004). Testador automático emétodo de avaliacão de programas em java. In XVI Salão de Iniciacão Científica e XIII Feira de Iniciacão Científica, Porto Alegre, RS.
Orcero, D. S. (2000). The code analyser LClint. Linux J., 2000(73).
Pieterse, V. (2013). Automated assessment of programming assignments. In Proceedings of the 3rd Computer Science Education Research Conference on Computer Science Education Research, CSERC ’13, pages 4:45–4:56, Open Univ., Heerlen, The Netherlands, The Netherlands. Open Universiteit, Heerlen.
Rahman, K. A., Ahmad, S., Nordin, M. J., and Maklumat, F. T. D. S. (2008). The Design of an Automated C Programming Assessment Using Pseudo-code Comparison Technique.
Rees, M. J. (1982). Automatic assessment aids for pascal programs. SIGPLAN Not., 17:33–42.
Romli, R., Sulaiman, S., and Zamli, K. (2010). Automatic programming assessment and test data generation a review on its approaches. In Information Technology (ITSim), 2010 International Symposium in, volume 3, pages 1186 –1192.
Saikkonen, R., Malmi, L., and Korhonen, A. (2001). Fully automatic assessment of programming exercises. SIGCSE Bull., 33:133–136.
Truong, N., Roe, P., and Bancroft, P. (2004). Static analysis of students’ java programs. In Proceedings of the Sixth Australasian Conference on Computing Education - Volume 30, ACE ’04, pages 317–325, Darlinghurst, Australia, Australia. Australian Computer Society, Inc.
Vujosevic-Janicic, M., Nikolic, M., Tosic, D., and Kuncak, V. (2013). Software verification and graph similarity for automated evaluation of students assignments. Information and Software Technology, 55(6):1004 – 1016.
Wang, T., Su, X., Ma, P., Wang, Y., and Wang, K. (2011). Ability-training-oriented automated assessment in introductory programming course. Comput. Educ., 56(1):220–226.
Wu,W., Li, G., Sun, Y.,Wang, J., and Lai, T. (2007). AnalyseC: A Framework for Assessing Students’ Programs at Structural and Semantic Level. In Control and Automation, 2007. ICCA 2007. IEEE International Conference on, pages 742 –747.
Xu, S. and Chee, Y. S. (2003). Transformation-based diagnosis of student programs for programming tutoring systems. IEEE Transactions on Software Engineering, 29:360–384.
Zin, A. and Foxley, E. (1994). Analyse:an automatic program assessment system. Malaysian Journal of Computer Science, 7:123–142.
Publicado
20/07/2015
Como Citar
DE OLIVEIRA, Márcia; OLIVEIRA, Elias.
Abordagens, Práticas e Desafios da Avaliação Automática de Exercícios de Programação. In: WORKSHOP DE DESAFIOS DA COMPUTAÇÃO APLICADA À EDUCAÇÃO (DESAFIE!), 4. , 2015, Recife.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2015
.
p. 131-140.
DOI: https://doi.org/10.5753/desafie.2015.10048.
