Language Model Architectures for Behavioral Assessment: A Case Study on MAPER

• Erik J. F. Nascimento UFC
• Jessyca A. Bessa IFCE
• Maryane C. Lima IFCE
• Igor R. S. Valente IFCE
• Saulo M. Maia UFC
• Maria L. R. Correa FGV

Resumo

Professional profile mapping tests (PPMT) are used to improve employees’ skills in organizations. With the increase in demand, professionals who perform these tests often become overloaded and saturated. Large language models (LLMs) have proven to be, when properly applied, excellent support tools for generating reports and diagnostics in several areas. In this work, we apply, for the first time in the literature, a LLM to perform personalized PPMT feedbacks. To achieve our results: we built a real database containing (inputs and outputs) PPMT feedback information; we built an optimal prompt for the task; and we fine-tuned a LLM with our database plus prompt. Our research focused on two forms of fine-tuning: i.e., Low-Rank Adaptation (LoRA) and Retrieval Augmented Generation (RAG). We evaluate our approach using popular metrics such as rouge 1 and 2, rougeL and BERTScore. We also compared the models in terms of generation time and resource consumption. Our experiments have shown that the RAG approach, was capable of generating semantically accurate feedback with a BERTScore precision of around 76%, but tends to be syntactically different from the expected pattern (i.e., that of the psychologist). The LoRA approach was closer both semantically and syntactically to the psychologist’s patter with a BERTScore precision of 89%.