Criteria for the Evaluation of Animated Pedagogical Agents: A Proposed Method

• Márcia Cristina Moraes PUCRS
• Milene Selbach Silveira PUCRS

Abstract

Although several projects are being developed in the area of pedagogical animated agents, there isn’t a method to describe criteria that ought to be considered to evaluate these agents. This paper proposes a method for pedagogical animated agents evaluation based on concepts provided by Artificial Intelligence, Computer in Education and Human-Computer Interaction.

References

Ball, G., Linf, D., Kurlander, D., Miller, J., Pugh, D., Skelly, T., Stankosky, A., Thiel, D., Van Dantzich, M., & Wax, T. (1997). “Lifelike Computer Characters: The Persona Project at Microsof”. In Bradshaw, J. M. (Ed), Software Agents. Menlo Park, CA: AAAI Press.

Bates, J. (1992). The Nature of Characters in Interactive Worlds and The OZ Project. (Tech. Rep. No. CMU-CS-92-200). Pittsburgh, California: Carnegie Mellon University, Computer Science Department.

Craig, S., Glohson, B., & Driscoll, D. (2002). Animated Pedagogical Agents in Multimedia Educational Environments: Effects of Agent Properties, Picture Features, and Redundancy. Journal of Educational Psychology , 94 (2), 428–434.

Christoph, N. (2004). “Empirical Evaluation Methodology for Embodied Conversational Agents”. In Ruttkay, Z., & Pelachaud, C. (Eds.), From Brows to Trust (pp.67-99). The Netherlands: Kluwer Academic Publishers.

Conati, C., & Zhao, X. (2004). Building and Evaluating an intelligent Pedagogical Agent to Improve the Effectiveness of an Educational Game. International Conference on Intelligent User Interfaces IUI'04 (pp.6-13). Madeira, Portugal: ACM Press.

Gerbhard, P., Kipp, M., Klesen, M., & Rist, T. (2003). Authoring Scenes for Adaptive, Interactive Performances. International Conference on Autonomous Agents 2003 (pp. 725-732). Melbourne: ACM Press.

Hartmann, B., Mancini, M., Buisine, S., & Pelachaud, C. (2005). Design and Evaluation of Expressive Gesture Synthesis for Embodied Conversational Agents. International Conference on Autonomous Agents AAMAS'05 (pp.1095-1096). Utrecht, Netherlands: ACM Press.

Hayes-Roth, B., Doyle, P. (1998). Animate Characters. Autonomous Agents and Multi-Agent Systems, 1, 195-230.

Hix, D., & Hartson H.R. (1993). Developing User Interfaces: ensuring usability through product & process. New York, NY: John Wiley & Sons.

Isbister, K., & Doyle, P. (2004). “The Blind Men and the Elephant Revisited: Evaluating Interdisciplinary ECA Research”. In Z. Ruttkay & C. Pelachaud (Eds.), From Brows to Trust: Evaluating Embodied Conversational Agents (pp. 3-26). Norwell, MA: Kluwer Academic Publishers.

Johnson, W. L., Rickel, J. W., & Lester, J. (2000) Animated Pedagogical Agents: Face-to-Face Interaction in Interactive Learning Environments. International Journal of Artificial Intelligence in Education, 11, 47-78.

Mayes, J.T., & Fowler, C.J. (1999). Learning Technology and Usability: a framework for understanding courseware. Interacting with Computers, 11, 485-497.

Moraes, M. C., Silveira, M. S. (2006). How am I? Guidelines for Animated Interface Agents Evaluation. IEEE/ACM International Conference on Intelligent Agent Technology (pp. 200-203). Hong Kong: ACM Press.

Mulken, S.V., André, E., & Müller, J. (1998). The Persona Effect: How Substantial Is It?. XIII Human Computer Interaction (pp.53-66). Berlin: Springer.

Nielsen, J. (1994). “Heuristic evaluation”. In Nielsen, J., & Mack, R.L. (Eds.), Usability Inspection Methods. New York, NY: John Wiley & Sons.

Paiva, A., Dias, J., Sobral, D., Aylett, R., Woods, S., Hall, l., & Zoll, C. (2005). Learning by Evoking Empathy with Synthetic Characters. Applied Artificial Intelligence, 19, 235-266.

Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction design: beyond human-computer interaction. New York, NY: John Wiley & Sons.

Prendinger, H., Ma, C., Yingzi, J., Kazutaka, K., & Ishizuka, M. (2005). Evaluating the Interaction with Synthetic Agents using Attention and Affect Tracking. International Conference on Autonomous Agents AAMAS'05 (pp.1099-1100). Utrecht, Netherlands: ACM Press.

Reeves, B., Nass, C. (1996). The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge: Cambridge University Press.

Rubin, J. (1994). Handbook of Usability Testing. New York, NY: John Wiley & Sons.

Ruttkay, Z., Dormann, C., & Noot, H. (2004). “Embodied Conversational Agents on a Common Ground: A Framework for Design and Evaluation”. In Z. Ruttkay & C. Pelachaud (Eds.), From Brows to Trust: Evaluating Embodied Conversational Agents (pp. 27-66). Norwell, MA: Kluwer Academic Publishers.

Shneiderman, B. (1998). Designing the User Interface: strategies for effective human-computer interaction. Reading, MA: Addison-Wesley, 1998.

Squires, D., & Preece, J. (1999). Predicting quality in educational software: evaluating for learning, usability and the synergy between them. Interacting with Computers, 11, 467-483.

W3C. (2006). Web Content Accessibility Guidelines 1.0. Retrieved December 15, 2006, from [link].