Large Language Models as Instrument-Aware Guardrails for Structured Mental Health Questionnaires

• Rafhael da S. Monteiro UFPA
• Carlos Augusto M. Oeiras UFPA
• Jefferson M. de Morais UFPA

DOI: https://doi.org/10.5753/eniac.2026.30019

Resumo

Psychometric instruments such as PHQ-9, GAD-7, and BDI-II are increasingly integrated into digital assessment systems, where preserving instrument-specific response constraints is essential for reliable data collection. This work investigates whether Large Language Models (LLMs) can operate as instrument-aware guardrails for validating responses to structured mental health questionnaires. A controlled synthetic dataset containing structurally valid and invalid responses was evaluated independently by GPT-4o-mini, DeepSeek-V3, and LLaMA3 using a common instrument-specific validation protocol. The models achieved substantial to almost-perfect inter-model agreement (Fleiss’ κ: 0.795–0.933), while a deterministic rule-based baseline achieved 1.000 accuracy across all three instruments. These findings indicate that the evaluated LLMs can consistently apply explicit instrument-specific validity criteria under controlled conditions. However, the perfect performance of the deterministic baseline indicates that the present benchmark does not establish semantic superiority of LLMs over rule-based validation. The results therefore provide evidence for the feasibility of LLM-based instrument-aware validation as a complementary layer in digital assessment systems, while motivating future evaluation with genuinely free-form and semantically variable responses.

Referências

Brickman, J., Gupta, M., and Oltmanns, J. R. (2025). Large language models for psychological assessment: A comprehensive overview. Advances in Methods and Practices in Psychological Science, 8(3).

Dong, Y. et al. (2024). Safeguarding large language models: A survey. arXiv preprint arXiv:2406.02622.

Eichstaedt, J. C., Smith, R. J., Merchant, R. M., Ungar, L. H., Crutchley, P., Preotiuc-Pietro, D., Asch, D. A., and Schwartz, H. A. (2018). Facebook language predicts depression in medical records. Proceedings of the National Academy of Sciences, 115(44):11203–11208.

Inan, H. et al. (2023). Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674.

Kjell, O. N. E., Kjell, K., and Schwartz, H. A. (2024). Beyond rating scales: With targeted evaluation, large language models are poised for psychological assessment. Psychiatry Research, 333:115667.

Kroenke, K., Spitzer, R. L., and Williams, J. B. W. (2001). The phq-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9):606–613.

McCoy, T. H., Castro, V. M., and Perlis, R. H. (2025). Estimating depression severity in narrative clinical notes using large language models. Journal of Affective Disorders, 381:270–274.

McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica, 22(3):276–282.

Moëll, B. and Sand Aronsson, F. (2026). High-accuracy prediction of mental health scores from english bert embeddings trained on llm-generated synthetic self-reports. Frontiers in Digital Health.

Nair, R. P., Thushara, M. G., and Sugumaran, V. (2025). Fine-tuned llms versus rule-based nlp for uml diagram generation: An educational evaluation. IEEE Access, 13:211605–211619.

Nguyen, H. H. et al. (2025). Foundation models as guardrails: Llm- and vlm-based approaches to safety and alignment. In Proceedings of the APSIPA Annual Summit and Conference.

Ohse, J. et al. (2024). Large language models for depression screening from transcribed semi-clinical interviews. Computer Speech & Language, 88:101663.

Spitzer, R. L., Kroenke, K., Williams, J. B. W., and Löwe, B. (2006). A brief measure for assessing generalized anxiety disorder: The gad-7. Archives of Internal Medicine, 166(10):1092–1097.

Teferra, B. G. et al. (2025). Leveraging large language models for automated depression screening. PLOS Digital Health, 4(7):e0000943.

Wang, Y.-P. and Gorenstein, C. (2013). Psychometric properties of the beck depression inventory-ii: A comprehensive review. Revista Brasileira de Psiquiatria, 35(4):416–431.