Evaluating Semantic Caching in Practice: A Study on a LLM-Driven Distributed Application in a Brazilian EdTech - Nóbrega, Henrique Lopes; Maia, David Candeia Medeiros; Brunet, João (Anais Estendidos do Simpósio Brasileiro de Redes de Computadores e Sistemas Distribuídos (SBRC), 2025)
Large Language Models (LLMs) support various business functions, such as Alura’s use of GPT-4 to assess students’ answers. However, high computational and financial costs, along with response time issues, limit scalability. While caching mechanisms offer an alternative, traditional cache fails to evaluate queries semantically, leading to low hit rates. This work evaluates semantic caching on an Alura’s dataset of 94, 913 answers from 20, 639 students. Results show that 45.1% of LLM requests could be served from the cache, significantly reducing costs and improving response times by 4–12× for cache hits.