KV-RAPTOR: Scalable Tree-Structured Retrieval with KV Cache Compression for Question-Answering Systems - J. da Silva, João Gabriel; de Oliveira, Sávio S. T.; Alves, Lucas Alexandria; Eiris, Nicolás; Galvão Filho, Arlindo R. (Anais do Simpósio Brasileiro de Banco de Dados (SBBD); 2025: Anais do XL Simpósio Brasileiro de Bancos de Dados, 2025)
This paper introduces KV-RAPTOR, a latency-optimized variant of the RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) pipeline for Retrieval-Augmented Generation (RAG). By integrating CacheGen, a compressed key-value (KV) cache reuse mechanism, into RAPTOR’s tree-based index, we demonstrate that it is possible to reduce generation latency without sacrificing answer quality. We evaluate our method on both English and Portuguese datasets, showing consistent reductions in time-to-first-token and end-to-end latency while preserving performance across diverse linguistic and retrieval contexts.