Abordagens, Práticas e Desafios da Avaliação Automática de Exercícios de Programação
Abstract
Automatic assessment has emerged as an important method to help teachers to correct exercises and enable immediate feedback for students, especially in large classes. Some proposals have evaluated items as correct execution of programs, coding, programming style, software metrics, plagiarism and even psychological aspects. However, despite the programming assessment systems have evolved, such systems are still limited to evaluate if educational goals really have been achieved. Thus, among many challenges, the main challenge of programming automatic assessment is to assess what is actually a teacher wants to evaluate. Aiming to discuss solutions to this challenge, this paper presents a review of programming automatic assessment and points out ways to make it a functional assessment in the next ten years.
References
Ala-Mutka, K. M. (2005). A survey of automated assessment approaches for programming assignments. Computer Science Education, 15(2):83–102.
Benford, S. D., Burke, E. K., Foxley, E., and Higgins, C. A. (1995). The Ceilidh system for the automatic grading of students on programming courses. In Proceedings of the 33rd annual on Southeast regional conference, ACM-SE 33, pages 176–182, New York, NY, USA. ACM.
Blumenstein, M. M., Green, S., Nguyen, A. T., and Muthukkumarasamy, V. (2004). Game: A generic automated marking environment for programming assessment.
Campos, C. and Ferreira, C. (2004). Boca: um sistema de apoio para competições de programação. In XII Workshop de Educação em Computação (WEI) - SBC 2004), Salvador, BA.
C.M. Tang, Y. Y. and Poon, C. (2009). An approach towards automatic testing of student programs using token patterns. In Proceedings of the 17th International Conference on Computers in Education, pages 118–190, Hong Kong.
Curtis, B., Sheppard, S. B.,Milliman, P., Borst,M. A., and Love, T. (1979). Measuring the psychological complexity of software maintenance tasks with the halstead and mccabe metrics. IEEE Trans. Softw. Eng., 5(2):96–104.
De Souza, D., Maldonado, J., and Barbosa, E. (2011). Progtest: An environment for the submission and evaluation of programming assignments based on testing activities. In Software Engineering Education and Training (CSEE T), 2011 24th IEEE-CS Conference on, pages 1 –10.
Dias, P. (2001). Avaliação automática de exercícios em sql. Master’s thesis, Faculdade de Engenharia da Universidade do Porto, Porto, Portugal.
Ebrahimi, A. (1994). Novice programmer errors: language constructs and plan composition. International Journal of Human-Computer Studies, 41(4):457 – 480.
Franca, A., Soares, J., Gomes, D., and G.C.Barroso (2011). Um sistema orientado a serviços para suporte a atividades de laboratório em disciplinas de técnicas de programacão com integracão ao ambiente moodle. RENOTE - Revista Novas Tecnologias na Educação, 9(1).
Gerdes, A., Jeuring, J. T., and Heeren, B. J. (2010). Using strategies for assessment of programming exercises. In Proceedings of the 41st ACM technical symposium on Computer science education, SIGCSE ’10, pages 441–445, New York, NY, USA. ACM.
Hollingsworth, J. (1960). Automatic graders for programming classes. Commun. ACM, 3(10):528–529.
Hung, S.-l., Kwok, L.-f., and Chung, A. (1993). New metrics for automated programming assessment. In Proceedings of the IFIP WG3.4/SEARCC (SRIG on Education and Training) Working Conference on Software Engineering Education, pages 233–243, Amsterdam, The Netherlands, The Netherlands. North-Holland Publishing Co.
Ihantola, P., Ahoniemi, T., Karavirta, V., and Seppälä, O. (2010). Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, Koli Calling ’10, pages 86–93, New York, NY, USA. ACM.
Jackson, D. and Usher, M. (1997). Grading student programs using assyst. SIGCSE Bull., 29(1):335–339.
Malmi, L., Korhonen, A., and Saikkonen, R. (2002). Experiences in automatic assessment on mass courses and issues for designing virtual courses. In Proceedings of the 7th annual conference on Innovation and technology in computer science education, ITiCSE ’02, pages 55–59, New York, NY, USA. ACM.
Moreira, M. P. and Favero, E. L. (2009). Um ambiente para ensino de programação com feedback automático de exercícios. In XVII Workshop Sobre Educação em Computação (WEI) - CSBC 2009.
Moussiades, L. and Vakali, A. (2005). Pdetect: A clustering approach for detecting plagiarism in source code datasets. The computer journal, 48(6):651–661.
Naude, K. A., Greyling, J. H., and Vogts, D. (2010). Marking student programs using graph similarity. Computers & Education, 54(2):545 – 561.
Nunes, I. and Lisboa,M. (2004). Testador automático emétodo de avaliacão de programas em java. In XVI Salão de Iniciacão Científica e XIII Feira de Iniciacão Científica, Porto Alegre, RS.
Orcero, D. S. (2000). The code analyser LClint. Linux J., 2000(73).
Pieterse, V. (2013). Automated assessment of programming assignments. In Proceedings of the 3rd Computer Science Education Research Conference on Computer Science Education Research, CSERC ’13, pages 4:45–4:56, Open Univ., Heerlen, The Netherlands, The Netherlands. Open Universiteit, Heerlen.
Rahman, K. A., Ahmad, S., Nordin, M. J., and Maklumat, F. T. D. S. (2008). The Design of an Automated C Programming Assessment Using Pseudo-code Comparison Technique.
Rees, M. J. (1982). Automatic assessment aids for pascal programs. SIGPLAN Not., 17:33–42.
Romli, R., Sulaiman, S., and Zamli, K. (2010). Automatic programming assessment and test data generation a review on its approaches. In Information Technology (ITSim), 2010 International Symposium in, volume 3, pages 1186 –1192.
Saikkonen, R., Malmi, L., and Korhonen, A. (2001). Fully automatic assessment of programming exercises. SIGCSE Bull., 33:133–136.
Truong, N., Roe, P., and Bancroft, P. (2004). Static analysis of students’ java programs. In Proceedings of the Sixth Australasian Conference on Computing Education - Volume 30, ACE ’04, pages 317–325, Darlinghurst, Australia, Australia. Australian Computer Society, Inc.
Vujosevic-Janicic, M., Nikolic, M., Tosic, D., and Kuncak, V. (2013). Software verification and graph similarity for automated evaluation of students assignments. Information and Software Technology, 55(6):1004 – 1016.
Wang, T., Su, X., Ma, P., Wang, Y., and Wang, K. (2011). Ability-training-oriented automated assessment in introductory programming course. Comput. Educ., 56(1):220–226.
Wu,W., Li, G., Sun, Y.,Wang, J., and Lai, T. (2007). AnalyseC: A Framework for Assessing Students’ Programs at Structural and Semantic Level. In Control and Automation, 2007. ICCA 2007. IEEE International Conference on, pages 742 –747.
Xu, S. and Chee, Y. S. (2003). Transformation-based diagnosis of student programs for programming tutoring systems. IEEE Transactions on Software Engineering, 29:360–384.
Zin, A. and Foxley, E. (1994). Analyse:an automatic program assessment system. Malaysian Journal of Computer Science, 7:123–142.
Benford, S. D., Burke, E. K., Foxley, E., and Higgins, C. A. (1995). The Ceilidh system for the automatic grading of students on programming courses. In Proceedings of the 33rd annual on Southeast regional conference, ACM-SE 33, pages 176–182, New York, NY, USA. ACM.
Blumenstein, M. M., Green, S., Nguyen, A. T., and Muthukkumarasamy, V. (2004). Game: A generic automated marking environment for programming assessment.
Campos, C. and Ferreira, C. (2004). Boca: um sistema de apoio para competições de programação. In XII Workshop de Educação em Computação (WEI) - SBC 2004), Salvador, BA.
C.M. Tang, Y. Y. and Poon, C. (2009). An approach towards automatic testing of student programs using token patterns. In Proceedings of the 17th International Conference on Computers in Education, pages 118–190, Hong Kong.
Curtis, B., Sheppard, S. B.,Milliman, P., Borst,M. A., and Love, T. (1979). Measuring the psychological complexity of software maintenance tasks with the halstead and mccabe metrics. IEEE Trans. Softw. Eng., 5(2):96–104.
De Souza, D., Maldonado, J., and Barbosa, E. (2011). Progtest: An environment for the submission and evaluation of programming assignments based on testing activities. In Software Engineering Education and Training (CSEE T), 2011 24th IEEE-CS Conference on, pages 1 –10.
Dias, P. (2001). Avaliação automática de exercícios em sql. Master’s thesis, Faculdade de Engenharia da Universidade do Porto, Porto, Portugal.
Ebrahimi, A. (1994). Novice programmer errors: language constructs and plan composition. International Journal of Human-Computer Studies, 41(4):457 – 480.
Franca, A., Soares, J., Gomes, D., and G.C.Barroso (2011). Um sistema orientado a serviços para suporte a atividades de laboratório em disciplinas de técnicas de programacão com integracão ao ambiente moodle. RENOTE - Revista Novas Tecnologias na Educação, 9(1).
Gerdes, A., Jeuring, J. T., and Heeren, B. J. (2010). Using strategies for assessment of programming exercises. In Proceedings of the 41st ACM technical symposium on Computer science education, SIGCSE ’10, pages 441–445, New York, NY, USA. ACM.
Hollingsworth, J. (1960). Automatic graders for programming classes. Commun. ACM, 3(10):528–529.
Hung, S.-l., Kwok, L.-f., and Chung, A. (1993). New metrics for automated programming assessment. In Proceedings of the IFIP WG3.4/SEARCC (SRIG on Education and Training) Working Conference on Software Engineering Education, pages 233–243, Amsterdam, The Netherlands, The Netherlands. North-Holland Publishing Co.
Ihantola, P., Ahoniemi, T., Karavirta, V., and Seppälä, O. (2010). Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, Koli Calling ’10, pages 86–93, New York, NY, USA. ACM.
Jackson, D. and Usher, M. (1997). Grading student programs using assyst. SIGCSE Bull., 29(1):335–339.
Malmi, L., Korhonen, A., and Saikkonen, R. (2002). Experiences in automatic assessment on mass courses and issues for designing virtual courses. In Proceedings of the 7th annual conference on Innovation and technology in computer science education, ITiCSE ’02, pages 55–59, New York, NY, USA. ACM.
Moreira, M. P. and Favero, E. L. (2009). Um ambiente para ensino de programação com feedback automático de exercícios. In XVII Workshop Sobre Educação em Computação (WEI) - CSBC 2009.
Moussiades, L. and Vakali, A. (2005). Pdetect: A clustering approach for detecting plagiarism in source code datasets. The computer journal, 48(6):651–661.
Naude, K. A., Greyling, J. H., and Vogts, D. (2010). Marking student programs using graph similarity. Computers & Education, 54(2):545 – 561.
Nunes, I. and Lisboa,M. (2004). Testador automático emétodo de avaliacão de programas em java. In XVI Salão de Iniciacão Científica e XIII Feira de Iniciacão Científica, Porto Alegre, RS.
Orcero, D. S. (2000). The code analyser LClint. Linux J., 2000(73).
Pieterse, V. (2013). Automated assessment of programming assignments. In Proceedings of the 3rd Computer Science Education Research Conference on Computer Science Education Research, CSERC ’13, pages 4:45–4:56, Open Univ., Heerlen, The Netherlands, The Netherlands. Open Universiteit, Heerlen.
Rahman, K. A., Ahmad, S., Nordin, M. J., and Maklumat, F. T. D. S. (2008). The Design of an Automated C Programming Assessment Using Pseudo-code Comparison Technique.
Rees, M. J. (1982). Automatic assessment aids for pascal programs. SIGPLAN Not., 17:33–42.
Romli, R., Sulaiman, S., and Zamli, K. (2010). Automatic programming assessment and test data generation a review on its approaches. In Information Technology (ITSim), 2010 International Symposium in, volume 3, pages 1186 –1192.
Saikkonen, R., Malmi, L., and Korhonen, A. (2001). Fully automatic assessment of programming exercises. SIGCSE Bull., 33:133–136.
Truong, N., Roe, P., and Bancroft, P. (2004). Static analysis of students’ java programs. In Proceedings of the Sixth Australasian Conference on Computing Education - Volume 30, ACE ’04, pages 317–325, Darlinghurst, Australia, Australia. Australian Computer Society, Inc.
Vujosevic-Janicic, M., Nikolic, M., Tosic, D., and Kuncak, V. (2013). Software verification and graph similarity for automated evaluation of students assignments. Information and Software Technology, 55(6):1004 – 1016.
Wang, T., Su, X., Ma, P., Wang, Y., and Wang, K. (2011). Ability-training-oriented automated assessment in introductory programming course. Comput. Educ., 56(1):220–226.
Wu,W., Li, G., Sun, Y.,Wang, J., and Lai, T. (2007). AnalyseC: A Framework for Assessing Students’ Programs at Structural and Semantic Level. In Control and Automation, 2007. ICCA 2007. IEEE International Conference on, pages 742 –747.
Xu, S. and Chee, Y. S. (2003). Transformation-based diagnosis of student programs for programming tutoring systems. IEEE Transactions on Software Engineering, 29:360–384.
Zin, A. and Foxley, E. (1994). Analyse:an automatic program assessment system. Malaysian Journal of Computer Science, 7:123–142.
Published
2015-07-20
How to Cite
DE OLIVEIRA, Márcia; OLIVEIRA, Elias.
Abordagens, Práticas e Desafios da Avaliação Automática de Exercícios de Programação. In: COMPUTER CHALLENGES WORKSHOP ON EDUCATION (DESAFIE!), 4. , 2015, Recife.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2015
.
p. 131-140.
DOI: https://doi.org/10.5753/desafie.2015.10048.
