Jean Piton-Gonçalves
ABSTRACT
Given a set of items, a Multidimensional Computer Adaptive Test (MCAT) selects those items from the bank according to the estimated abilities of the student, resulting in an individualized test. MCATs seek to maximize the test's accuracy, based on multiple simultaneous examination abilities (unlike a Computer Adaptive Test - CAT - which evaluates a single ability) using the sequence of items previously answered. Although MCATs have been very well studied from a statistical point of view, there is no computational system that covers all the steps needed for its appropriated use such as: the use of a calibrated item bank, proposal of initial and stopping criteria for the test, criteria for estimating the ability of the examinee and criteria to select items. The purpose of this paper is twofold: (i) to present an innovative architecture of an MCAT for real users, as a Web application, and (ii) to discuss the theoretical and methodological development of such MCAT, through a new approach named here Computer-based Multidimensional Adaptive Testing (CBMAT). The proof of concept of CBMAT was an implementation called Multidimensional Adaptive Test System for Educational Purposes (MADEPT). In simulations, MADEPT proved to be a computer system suitable for applications with real users, secure, accurate and portable.
- S. M. Alu1sio, V. T. de Aquino, R. Pizzirani, and O. N. Oliveira. High order skills with partial knowledge evaluation: Lessons learned from using a computer-based proficiency test of English for academic purposes. Journal of Information Technology Education, 2(1):185--201, 2003.Google Scholar
- D. F. Andrade, H. R. Tavares, and R. da Cunha Valle. Teoria de Resposta ao Item: Conceitos e Aplicacoes. Associacao Brasileira de Estat1stica, Sao Paulo, 2000.Google Scholar
- D. Bartram. Computer-Based Testing and the Internet: Issues and Advances, chapter Testing on the Internet: Issues, Challenges and Opportunities in the Field of Occupational Assessment, pages 13--37. John Wiley and Sons, Ltd, 2006.Google Scholar
- R. D. Bock, R. Gibbons, and E. Muraki. Full-information item factor analysis. Applied Psychological Measurement, 12(3):261--280, 1988.Google ScholarCross Ref
- J. E. Bruno. Admissible probability measures in instructional management. Journal of Computer-Based Instruction, 14, 1987. Google ScholarDigital Library
- H.-H. Chang and Z. Ying. A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20:213--229, 1996.Google ScholarCross Ref
- M. Curi, J. Piton-Gonçalves, T. A. M. Ricarte, and S. M. Alu1sio. Metodos de selecao de itens em teste adaptativo multidimensional. 1o Congresso Brasileiro de Teoria de Resposta ao Item.Google Scholar
- Q. Diao and M. D. Reckase. Comparison of ability estimation and item selection mathods in multidimensional computerized adaptive testing. In Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing, 2009.Google Scholar
- E. Guzman, R. Conejo, and E. Garc1a-hervas. An authoring environment for adaptive testing. Educational Technology & Society, 8:66--76, 2005.Google Scholar
- S. Kullback. Information theory and statistics. John Wiley and Sons, 1959.Google Scholar
- S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79--86, 1951.Google ScholarCross Ref
- Y. H. Li and R. W. Lissitz. An evaluation of the accuracy of multidimensional irt linking. Applied Psychological Measurement, 24:115--138, 2000.Google ScholarCross Ref
- J. M. Linacre. Computer-adaptive testing: a methodology whose time has come. Published in Sunhee Chae, Unson Kang, Eunhwa Jeon and J. M. Linacre. Development of computerised middle school achievement test, 2000.Google Scholar
- F. M. Lord. Application of Item Response Theory to Practical Testing Problems. Lawrence Erlbaum Associates, Hilsdale, New Jersey, EUA, first edition, 1980.Google Scholar
- C. C. Marinagi, V. G. Kaburlasos, and V. T. Tsoukalas. An architecture for an adaptive assessment tool. Frontiers In Education Conference - Global Engineering: Knowledge Without Borders, Opportunities Without Passports. FIE '07, (37th Annual):T3D-11-T3D-16, 2007.Google ScholarCross Ref
- C. McKenna and J. Bull. Design effective objective test questions: an introductory workshop. Proceedings of the Conference at Loughborough University, Flexible Learning, (Third):253--257, June 1999.Google Scholar
- J. Mulder and W. J. van der Linden. Elements of Adaptive Testing, chapter Multidimensional Adaptive Testing with Kullback-Leibler Information Item Selection, pages 77--101. Springer, 2010.Google Scholar
- J. Olea, V. Ponsoda, and G. Prieto. Tests Informatizados Fundamentos y Aplicaciones. Ediciones Piramede, 1999.Google Scholar
- C. G. Parshall, J. A. Spray, J. C. Kalohn, and T. Davey. Practical Considerations in Computer-Based Testing. Springer-Verlag, New York, Inc., 2002.Google Scholar
- M. Phankokkruad and K. Woraratpanya. Web service architecture for computer-adaptive testing on e-learning. Proceedings of World Academy of Science Engineering Technology, 46:347--351, 2008.Google Scholar
- J. Piton-Goncalves, S. M. Alu1sio, L. H. Mendonca, and O. O. Novaes. A learning environment for English for academic purposes based on adaptive tests and task-based systems. In J. Lester, R. Vicari, and F. Paraguacu, editors, Intelligent Tutoring Systems, volume 3220 of Lecture Notes in Computer Science, pages 1--11. Springer Berlin Heidelberg, 2004.Google Scholar
- J. Piton-Goncalves, M. Curi, and S. M. Alu1sio. Studies in kullback-leibler between subsequent posteriors: defining an information-based stopping critetion. Psychometrika: New York: Springer (submitted), 2012.Google Scholar
- J. Piton-Goncalves, A. J. Monzon, and S. M. Alu1sio. Metodos de avaliacao informatizada que tratam o conhecimento parcial do aluno e geram provas individualizadas. In Anais do XX Simposio Brasileiro de Informatica na Educacao, volume 1, Porto Alegre, RS, Novembro 2009. Sociedade Brasileira de Computacao.Google Scholar
- M. D. Reckase. An interactive computer program for tailored testing based on the oneparameter logistic model. Behaviour Research Methods and Instrumentation, 6(2):208--307, 1974.Google ScholarCross Ref
- M. D. Reckase. The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9(4):401--412, 1985.Google ScholarCross Ref
- M. D. Reckase. Multidimensional Item Response Theory. Springer: New York - NY, 2009. Google ScholarDigital Library
- M. D. Reckase and R. L. McKinely. The discriminating power of items that measure more than one dimension. Applied Psychological Measurement, 15:361--373, 1991.Google ScholarCross Ref
- P. Salcedo, M. Pinninghoff, and R. Contreras. Computerized adaptive tests and item response theory on a distance education platform. In J. Mira and J. alvarez, editors, Artificial Intelligence and Knowledge Engineering Applications: A Bioinspired Approach, volume 3562 of Lecture Notes in Computer Science, pages 22--29. Springer Berlin / Heidelberg, 2005. Google ScholarDigital Library
- Sao-Paulo. Matrizes de Referencia para a Avaliacao SARESP 2009, volume 1. Secretaria da Educacao do Governo do Estado de Sao Paulo, 2009.Google Scholar
- D. O. Segall. Multidimensional adaptive testing. Psychometrika, 61(2):331--354, June 1996.Google ScholarCross Ref
- D. O. Segall. Computerized Adaptive Testing: Theory and Practice, chapter Principles fo Multidimensional Adaptive Testing, pages 53--73. Kluwer Academic Publishers - New York, Boston, Dordrecht, London, Moscow, 2000.Google Scholar
- L. J. Simms and L. A. Clark. Validation of a computerized adaptive version for nonadaptive and adaptive personality (snap). Psychological Assessment, 17(1):28--43, 2005.Google ScholarCross Ref
- B. P. Veldkamp and W. J. van der Linden. Multidimensional adaptive testing with constraints on the test content. Psychometrika, 67(4):575--588, 2002.Google ScholarCross Ref
- C. Wang and H.-H. Chang. Item selection in multidimensional computerized adaptive testing-gaining information from different angles. Psychometrika, 76(3):363--384, 2011.Google ScholarCross Ref
- C. Wang, H.-H. Chang, and K. A. Boughton. Kullback-leibler information and its applications in multi-dimensional adaptive testing. Psychometrika, 76(1):13--19, 2011.Google ScholarCross Ref
- D. J. Weiss. Adaptive testing by computer. Journal of Consulting and Clinical Psychology, 53(6):774--789, 1985.Google ScholarCross Ref
- D. J. Weiss and G. G. Kingsbury. Application of computerized adaptive testing to educational problems. Journal of Education Measurement, 21:361--375, 1984.Google ScholarCross Ref
- L. A. M. Zaina, J. F. R. Jr., M. A. C. D. A. Cardieri, and G. Bressan. Adaptive learning in the educational e-lors system: an approach based on preference categories. IJLT, 6(4):341--361, 2011. Google ScholarDigital Library
Index Terms
- An architecture for multidimensional computer adaptive test with educational purposes
Recommendations
Evaluating computer-based and paper-based versions of an English-language listening test
This paper describes an English language listening test intended as computer-based testing material for secondary school students in Hong Kong, where considerable attention is being invested in online and computer-based testing. As well as providing a ...
Effects of computer-based testing on test performance and testing motivation
Computer-based testing (CBT) is a green computing strategies used to reduce paper consumption. However, some scholars have questioned the effectiveness of CBT and suggested for conducting systematic studies on CBT to carefully check its reliability and ...
Measuring Effects of Modality on Perceived Test Anxiety for Computer Programming Exams
SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science EducationThe modality students use to complete programming-centric exams is shifting from paper to computer. We developed and distributed a survey to investigate how exam modality affects student perceptions of test anxiety and performance during programming-...
Comments