A Synthetic Benchmark for Integrated Security Assessment of RAG-Based Corporate Assistants - Figueira, Larissa de Oliveira; Duarte, Luiz Otavio; Silva, Rafael Lucas (Anais Estendidos do Simpósio Brasileiro de Cibersegurança (SBSeg), 2026)
This paper presents a synthetic benchmark for integrated security assessment of RAG-based corporate assistants. The benchmark models a fictitious organization and instruments security-relevant stages across the RAG pipeline. It comprises 473 cases per configuration, including 352 adversarial cases organized into 23 attack families, and evaluates ten defensive configurations. The analysis combines family-macro Attack Success Rate with stage-level metrics to distinguish internal pipeline violations from failures observable in the final response. Results show that output-level mitigations may suppress visible leakage while leaving upstream authorization and retrieval failures unresolved, whereas layered observable defenses provide broader protection while preserving benign utility. An exploratory comparison with a local Llama model shows the same directional defensive pattern under a different generation mechanism. These findings support pipeline-level evaluation as a more informative approach to assessing RAG-based corporate assistants than final-response analysis alone.