Comparative Evaluation of Prompt Compression Techniques for Large Language Models: Extractive, Abstractive, and Token-Pruning

• Carolina Suyungütmüs UFCG
• Claudio E. C. Campelo UFCG

DOI: https://doi.org/10.5753/eniac.2026.30057

Resumo

Large Language Models (LLMs) face significant challenges when processing long prompts regarding computational cost and response quality. This paper presents a comparative evaluation of prompt compression methods, including an exploratory hybrid approach that combines reranking and semantic clustering. Experiments with Llama 3.1 8B Instruct and GPT-4o Mini on LongBench v2 show that accuracy differences among the evaluated methods were generally small relative to the uncertainty of the evaluation, with the standalone reranker achieving the highest observed accuracy at the 8k budget. Extractive methods showed the most favorable accuracy-latency trade-off among the evaluated approaches.

Referências

Ali, M. A., Li, Z., Yang, S., Cheng, K., Cao, Y., Huang, T., Hu, L., Yu, L., and Wang, D. (2024). Prompt-saw: Leveraging relation-aware graphs for textual prompt compression. arXiv preprint arXiv:2404.00489.

Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., and Li, J. (2023). Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.

Bai, Y., Tu, S., Zhang, J., Peng, H., Wang, X., Lv, X., and Li, J. (2024). Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

He, P., Liu, X., Gao, J., and Chen, W. (2020). Deberta: Decoding enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.

Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. (2024). Kvquant: Towards 10 million context length llm inference with kv cache quantization. arXiv preprint arXiv:2401.18079.

Jha, S., Erdogan, L. E., Kim, S., Keutzer, K., and Gholami, A. (2024). Characterizing prompt compression methods for long context inference. arXiv preprint arXiv:2407.08892.

Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M. A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sablayrolles, W. E. (2023a). Mistral 7b.

Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., and Qiu, L. (2023b). LLMLingua: Compressing prompts for accelerated inference of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376. Association for Computational Linguistics.

Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C. Y., Yang, Y., and Qiu, L. (2023c). Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839.

Kim, S., Hooper, C., Wattanawong, T., Kang, M., Yan, R., Genc, H., Dinh, G., Huang, Q., Keutzer, K., Mahoney, M. W., Shao, Y. S., and Gholami, A. (2023). Full stack optimization of transformer inference: a survey.

Kummer, C., Jurkschat, L., Färber, M., and Vahdati, S. (2026). Prompt compression in the wild: Measuring latency, rate adherence, and quality for faster llm inference. In European Conference on Information Retrieval, pages 257–271. Springer.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kuttler, H., and Lewis, M. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. ArXiv, abs/2005.11401.

Li, Y., Dong, B., Lin, C., and Guerin, F. (2023). Compressing context to enhance inference efficiency of large language models. arXiv preprint arXiv:2310.06201.

Li, Z., Liu, Y., Su, Y., and Collier, N. (2025). Prompt compression for large language models: A survey. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7182–7195.

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv., 55(9).

META AI (2025). Llama 3.1 8b instruct. [link]. Accessed: 2025-03-15.

Nogueira, R. and Cho, K. (2019). Passage re-ranking with bert. arXiv preprint arXiv:1901.04085.

Novikova, J., Dušek, O., Curry, A. C., and Rieser, V. (2017). Why we need new evaluation metrics for nlg. arXiv preprint arXiv:1707.06875.

OPENAI (2025). Gpt-4o mini: Advancing cost-efficient intelligence. [link]. Accessed: 2025-03-15.

Pradeep, R., Sharifymoghaddam, S., and Lin, J. (2023a). Rankvicuna: Zero-shot list-wise document reranking with open-source large language models. arXiv preprint arXiv:2309.15088.

Pradeep, R., Sharifymoghaddam, S., and Lin, J. (2023b). Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze! arXiv preprint arXiv:2312.02724.

Rousseeuw, P. J. (1987). Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53–65.

Shakir, A., Koenig, D., Lipp, J., and Lee, S. (2024). Boost your search with the crispy mixedbread rerank models.

Wolf, T. and Debut, L. (2020). Transformers: State-of-the-art natural language processing. [link].

Xiao, S., Liu, Z., Zhang, P., and Muennighoff, N. (2023). C-pack: Packaged resources to advance general chinese embedding.

Xu, F., Shi, W., and Choi, E. (2023). Recomp: Improving retrieval augmented lms with compression and selective augmentation. arXiv preprint arXiv:2310.04408.

Zhang, Z., Li, J., Lan, Y., Wang, X., and Wang, H. (2025). An empirical study on prompt compression for large language models. arXiv preprint arXiv:2505.00019.