Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime - Abonizio, Hugo; Almeida, Thales; Lotufo, Roberto; Nogueira, Rodrigo (Anais do Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), 2025)
Large language models (LLMs) often require massive corpora to effectively acquire new knowledge, yet updating a model with only thousands – or even a few million – tokens remains difficult. In this work, we study how to inject such small, unstructured knowledge and how this affects catastrophic forgetting. Using a recent-news corpus, we probe models with question-answer pairs to assess knowledge acquisition. Alongside a continued-pre-training baseline, we evaluate data augmentation strategies that create synthetic variations. Our results show that continued pre-training alone yields modest gains, whereas diverse rephrasings significantly boost learning – especially when prompts maximize variability. We also evaluate forgetting in this low-data regime and find that retrieval-augmented generation (RAG) updates, though effective, degrade out-of-domain performance more than parametric approaches. Finally, we show that the model can produce high-quality synthetic data itself, pointing toward self-improving updates. Code and data are available at https://github.com/hugoabonizio/knowledge-injection-methods/ .