A Goal-Oriented Chat-Like System for Evaluation of Large Language Models - Teodoro Junior, Guilherme S.; Peres, Sarajane M.; Fantinato, Marcelo; Brandão, Anarosa A. F.; Cozman, Fabio G. (Anais do Encontro Nacional de Inteligência Artificial e Computacional (ENIAC); 2024: Anais do XXI Encontro Nacional de Inteligência Artificial e Computacional, 2024)
Large language models have changed the way various applications are developed. Interactions with large language models have reached a new level of complexity and now act as real problem solvers. However, despite their apparent competence, it is still necessary to accredit them with respect to the tasks they are assigned. In this paper, we discuss a systemic approach to accredit large language models through their integration with a goal-oriented chat-like system. An experiment involving prompt engineering for two models from the GPT family illustrates our evaluation scheme when applied to a real-world chatbot use case; our evaluation scheme reveals, that the resulting chatbots perform well but are not yet ready for real-world dialogues under specific requirements.