Avaliando a Estratificação de Memória na Inferência de LLMs somente em CPU
Resumo
A capacidade de DRAM representa um fator limitante de custo e viabilidade para a execução da inferência com Grandes Modelos de Linguagem (LLMs) em ambientes sem aceleradores. Este trabalho investiga o impacto da alocação seletiva dos três principais componentes internos de um LLM (pesos de atenção, pesos da FFN e KV-Cache) entre DRAM local e uma camada de memória de maior capacidade e maior latência, em uma plataforma exclusivamente baseada em CPU. Utilizando o modelo LLaMA 2 7B quantizado em Q4 K M, avaliam-se oito estratégias de posicionamento em cinco cenários de uso que variam o tamanho do contexto e a proporção entre prefill e decode. Os resultados demonstram que não existe um componente universalmente mais tolerante à estratificação: a hierarquia de custos rotaciona ao longo do gradiente de contexto, sendo a KV-Cache o componente mais seguro sob contexto efetivo curto (−3,9% de degradação de decode) e o mais custoso sob contexto efetivo longo (até −46,8%). A decisão de alocação é formalizada como um problema bi-objetivo (DRAM liberada vs. vazão) resolvido por meio da fronteira de Pareto com um piso de aceitabilidade de vazão, do qual derivam-se diretrizes práticas de alocação sensíveis ao contexto.Referências
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. (2023). GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4895–4901, Singapore. Association for Computational Linguistics.
Alizadeh, K., Mirzadeh, I., Belenko, D., Khatamifard, S. K., Cho, M., Del Mundo, C. C., Rastegari, M., and Farajtabar, M. (2024). LLM in a flash: Efficient large language model inference with limited memory. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, pages 12562–12584. arXiv:2312.11514.
Baldassin, A., Barreto, J. a., Castro, D., and Romano, P. (2021). Persistent memory: A survey of programming support and implementations. ACM Comput. Surv., 54(7).
Das Sharma, D., Blankenship, R., and Berger, D. (2024). An introduction to the compute express link (cxl) interconnect. ACM Comput. Surv., 56(11).
Du, H., Wu, S., Kharlamova, A., Guan, N., and Xue, C. J. (2025). FlexInfer: Breaking memory constraint via flexible and efficient offloading for on-device LLM inference. In Proceedings of the 5th Workshop on Machine Learning and Systems (EuroMLSys), pages 56–65. ACM. arXiv:2503.03777.
Fang, Y., Xie, R., Haq, A. U., Ma, L., El Maghraoui, K., Wang, N., Wang, M., Liu, L., and Zhang, T. (2025). Accelerating LLM inference via dynamic KV cache placement in heterogeneous memory system. arXiv preprint arXiv:2508.13231.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2023). OPTQ: Accurate quantization for generative pre-trained transformers. In Proceedings of the 11th International Conference on Learning Representations (ICLR). Originally titled GPTQ on arXiv (arXiv:2210.17323). Renamed to OPTQ at camera-ready.
Ganjihal, S. R. (2025). Predictive multi-tier memory management for KV cache in large-scale GPU inference. arXiv preprint arXiv:2604.26968.
Gerganov, G. et al. (2023). llama.cpp. [link]. Accessed: 2025.
Geva, M., Schuster, R., Berant, J., and Levy, O. (2021). Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Gu, Y., Khadem, A., Umesh, S., Liang, N., Servot, X., Mutlu, O., Iyer, R., and Das, R. (2025). PIM is all you need: A CXL-enabled GPU-free system for large language model inference. In Proceedings of the 30th ACM International Conference on Ar chitectural Support for Programming Languages and Operating Systems (ASPLOS). arXiv:2502.07578.
Jiang, X., Zhou, Y., Cao, S., Stoica, I., and Yu, M. (2025). NEO: Saving GPU memory crisis with CPU offloading for online LLM inference. In Proceedings of Machine Learning and Systems (MLSys).
Lee, D., Kim, Y., Seshadri, V., Liu, J., Subramanian, L., and Mutlu, O. (2016). Simultaneous Multi-Layer Access: Improving 3D-Stacked Memory Bandwidth at Low Cost. ACM Transactions on Architecture and Code Optimization, 12(4):49:1–49:29.
Lee, T., Monga, S. K., Min, C., and Eom, Y. I. (2023). Memtis: Efficient memory tiering with dynamic page classification and page size determination. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP ’23). ACM.
Leviathan, Y., Kalman, M., and Matias, Y. (2023). Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 19274–19286.
Lin, J., Mi, J., Hong, Z., Wang, H., Liu, Q., Zhang, H., Li, P., and Guo, S. (2025a). KVDrive: A holistic multi-tier KV cache management system for long-context LLM inference. arXiv preprint arXiv:2605.18071.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. (2024). AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of the 7th Conference on Machine Learning and Systems (MLSys). Best Paper Award. arXiv:2306.00978.
Lin, M., Wang, X., Cox, G., Li, D., and Jeon, H. (2025b). HybridGen: Efficient LLM generative inference via CPU–GPU hybrid computing. arXiv preprint arXiv:2604.18529.
Liu, L., Zhao, S., Zhou, Y., He, Y., Wang, M., Han, Y., and Wang, Y. (2025). PAM: Processing across memory hierarchy for efficient KV-centric LLM serving system. arXiv preprint arXiv:2602.11521.
Luo, C., Cai, Z., Sun, H., Xiao, J., Yuan, B., Xiao, W., Hu, J., Zhao, J., Chen, B., and Anandkumar, A. (2025). HeadInfer: Memory-efficient LLM inference by head-wise offloading. arXiv preprint arXiv:2502.12574.
Maruf, H. A., Wang, H., Dhanotia, A., Weiner, J., Agarwal, N., Bhattacharya, P., Petersen, C., Chowdhury, M., Kanaujia, S., and Chauhan, P. (2023). Tpp: Transparent page placement for cxl-enabled tiered-memory. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, page 742–755.
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., and Jia, Z. (2024). Towards efficient generative large language model serving: A survey from algorithms to systems. arXiv preprint arXiv:2312.15234.
Na, S., Jeong, G., Ahn, B. H., Jezghani, A., Young, J., Hughes, C. J., Krishna, T., and Kim, H. (2025). FlexInfer: Flexible LLM inference with CPU computations. In Proceedings of the 8th Conference on Machine Learning and Systems (MLSys).
Shazeer, N. (2019). Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150.
Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. (2023). FlexGen: High-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of PMLR, pages 31094–31116.
Tang, Y., Cheng, R., Zhou, P., Liu, T., Liu, F., Tang, W., Bae, K., Chen, J., Xiang, W., and Shi, R. (2024). Exploring CXL-based KV cache storage for LLM serving. In NeurIPS 2024 Workshop on Machine Learning for Systems.
Touvron, H., Martin, L., Stone, K., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. (2024). Efficient streaming language models with attention sinks. In Proceedings of the 12th International Conference on Learning Representations (ICLR). arXiv:2309.17453.
Yang, J., Kim, J., Hoseinzadeh, M., Izraelevitz, J., and Swanson, S. (2020). An empirical guide to the behavior and use of scalable persistent memory. In 18th USENIX Conference on File and Storage Technologies (FAST ’20), pages 169–182, Santa Clara, CA. USENIX Association.
Yang, X., Hu, Q., Li, J., Li, F., Zhu, Y., Zhou, Y., Lin, Q., Dai, J., Kong, Y., Zhang, J., Xu, G., and Liu, Q. (2025). Beluga: A CXL-based memory architecture for scalable and efficient LLM KVCache management. arXiv preprint arXiv:2511.20172.
Yuan, Z., Shang, Y., Zhou, Y., Dong, Z., Zhou, Z., Xue, C., Wu, B., Li, Z., Gu, Q., Lee, Y. J., Yan, Y., Chen, B., Sun, G., and Keutzer, K. (2024). LLM inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402.16363.
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. (2023). H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems 36 (NeurIPS).
Zheng, X., Wang, Z., Ma, R., Wang, R., Wang, X., Chen, R., Zhang, P., Pan, S., Huang, Z., Wu, C., Zhang, Y., Cai, B., Liu, K., Ma, T., Du, Y., Deng, D., Wu, S., Zhu, G., Zhang, W., and Li, F. (2025). Adaptive multi-objective tiered storage configuration for KV cache in LLM service. arXiv preprint arXiv:2603.08739.
Zhong, Y., Berger, D. S., Waldspurger, C., Wee, R., Agarwal, I., Agarwal, R., Hady, F., Kumar, K., Hill, M. D., Chowdhury, M., and Cidon, A. (2024). Managing memory tiers with CXL in virtualized environments. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 37–56, Santa Clara, CA. USENIX Association.
Alizadeh, K., Mirzadeh, I., Belenko, D., Khatamifard, S. K., Cho, M., Del Mundo, C. C., Rastegari, M., and Farajtabar, M. (2024). LLM in a flash: Efficient large language model inference with limited memory. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, pages 12562–12584. arXiv:2312.11514.
Baldassin, A., Barreto, J. a., Castro, D., and Romano, P. (2021). Persistent memory: A survey of programming support and implementations. ACM Comput. Surv., 54(7).
Das Sharma, D., Blankenship, R., and Berger, D. (2024). An introduction to the compute express link (cxl) interconnect. ACM Comput. Surv., 56(11).
Du, H., Wu, S., Kharlamova, A., Guan, N., and Xue, C. J. (2025). FlexInfer: Breaking memory constraint via flexible and efficient offloading for on-device LLM inference. In Proceedings of the 5th Workshop on Machine Learning and Systems (EuroMLSys), pages 56–65. ACM. arXiv:2503.03777.
Fang, Y., Xie, R., Haq, A. U., Ma, L., El Maghraoui, K., Wang, N., Wang, M., Liu, L., and Zhang, T. (2025). Accelerating LLM inference via dynamic KV cache placement in heterogeneous memory system. arXiv preprint arXiv:2508.13231.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2023). OPTQ: Accurate quantization for generative pre-trained transformers. In Proceedings of the 11th International Conference on Learning Representations (ICLR). Originally titled GPTQ on arXiv (arXiv:2210.17323). Renamed to OPTQ at camera-ready.
Ganjihal, S. R. (2025). Predictive multi-tier memory management for KV cache in large-scale GPU inference. arXiv preprint arXiv:2604.26968.
Gerganov, G. et al. (2023). llama.cpp. [link]. Accessed: 2025.
Geva, M., Schuster, R., Berant, J., and Levy, O. (2021). Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Gu, Y., Khadem, A., Umesh, S., Liang, N., Servot, X., Mutlu, O., Iyer, R., and Das, R. (2025). PIM is all you need: A CXL-enabled GPU-free system for large language model inference. In Proceedings of the 30th ACM International Conference on Ar chitectural Support for Programming Languages and Operating Systems (ASPLOS). arXiv:2502.07578.
Jiang, X., Zhou, Y., Cao, S., Stoica, I., and Yu, M. (2025). NEO: Saving GPU memory crisis with CPU offloading for online LLM inference. In Proceedings of Machine Learning and Systems (MLSys).
Lee, D., Kim, Y., Seshadri, V., Liu, J., Subramanian, L., and Mutlu, O. (2016). Simultaneous Multi-Layer Access: Improving 3D-Stacked Memory Bandwidth at Low Cost. ACM Transactions on Architecture and Code Optimization, 12(4):49:1–49:29.
Lee, T., Monga, S. K., Min, C., and Eom, Y. I. (2023). Memtis: Efficient memory tiering with dynamic page classification and page size determination. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP ’23). ACM.
Leviathan, Y., Kalman, M., and Matias, Y. (2023). Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 19274–19286.
Lin, J., Mi, J., Hong, Z., Wang, H., Liu, Q., Zhang, H., Li, P., and Guo, S. (2025a). KVDrive: A holistic multi-tier KV cache management system for long-context LLM inference. arXiv preprint arXiv:2605.18071.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. (2024). AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. In Proceedings of the 7th Conference on Machine Learning and Systems (MLSys). Best Paper Award. arXiv:2306.00978.
Lin, M., Wang, X., Cox, G., Li, D., and Jeon, H. (2025b). HybridGen: Efficient LLM generative inference via CPU–GPU hybrid computing. arXiv preprint arXiv:2604.18529.
Liu, L., Zhao, S., Zhou, Y., He, Y., Wang, M., Han, Y., and Wang, Y. (2025). PAM: Processing across memory hierarchy for efficient KV-centric LLM serving system. arXiv preprint arXiv:2602.11521.
Luo, C., Cai, Z., Sun, H., Xiao, J., Yuan, B., Xiao, W., Hu, J., Zhao, J., Chen, B., and Anandkumar, A. (2025). HeadInfer: Memory-efficient LLM inference by head-wise offloading. arXiv preprint arXiv:2502.12574.
Maruf, H. A., Wang, H., Dhanotia, A., Weiner, J., Agarwal, N., Bhattacharya, P., Petersen, C., Chowdhury, M., Kanaujia, S., and Chauhan, P. (2023). Tpp: Transparent page placement for cxl-enabled tiered-memory. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, page 742–755.
Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., and Jia, Z. (2024). Towards efficient generative large language model serving: A survey from algorithms to systems. arXiv preprint arXiv:2312.15234.
Na, S., Jeong, G., Ahn, B. H., Jezghani, A., Young, J., Hughes, C. J., Krishna, T., and Kim, H. (2025). FlexInfer: Flexible LLM inference with CPU computations. In Proceedings of the 8th Conference on Machine Learning and Systems (MLSys).
Shazeer, N. (2019). Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150.
Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. (2023). FlexGen: High-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of PMLR, pages 31094–31116.
Tang, Y., Cheng, R., Zhou, P., Liu, T., Liu, F., Tang, W., Bae, K., Chen, J., Xiang, W., and Shi, R. (2024). Exploring CXL-based KV cache storage for LLM serving. In NeurIPS 2024 Workshop on Machine Learning for Systems.
Touvron, H., Martin, L., Stone, K., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. (2024). Efficient streaming language models with attention sinks. In Proceedings of the 12th International Conference on Learning Representations (ICLR). arXiv:2309.17453.
Yang, J., Kim, J., Hoseinzadeh, M., Izraelevitz, J., and Swanson, S. (2020). An empirical guide to the behavior and use of scalable persistent memory. In 18th USENIX Conference on File and Storage Technologies (FAST ’20), pages 169–182, Santa Clara, CA. USENIX Association.
Yang, X., Hu, Q., Li, J., Li, F., Zhu, Y., Zhou, Y., Lin, Q., Dai, J., Kong, Y., Zhang, J., Xu, G., and Liu, Q. (2025). Beluga: A CXL-based memory architecture for scalable and efficient LLM KVCache management. arXiv preprint arXiv:2511.20172.
Yuan, Z., Shang, Y., Zhou, Y., Dong, Z., Zhou, Z., Xue, C., Wu, B., Li, Z., Gu, Q., Lee, Y. J., Yan, Y., Chen, B., Sun, G., and Keutzer, K. (2024). LLM inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402.16363.
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. (2023). H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems 36 (NeurIPS).
Zheng, X., Wang, Z., Ma, R., Wang, R., Wang, X., Chen, R., Zhang, P., Pan, S., Huang, Z., Wu, C., Zhang, Y., Cai, B., Liu, K., Ma, T., Du, Y., Deng, D., Wu, S., Zhu, G., Zhang, W., and Li, F. (2025). Adaptive multi-objective tiered storage configuration for KV cache in LLM service. arXiv preprint arXiv:2603.08739.
Zhong, Y., Berger, D. S., Waldspurger, C., Wee, R., Agarwal, I., Agarwal, R., Hady, F., Kumar, K., Hill, M. D., Chowdhury, M., and Cidon, A. (2024). Managing memory tiers with CXL in virtualized environments. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 37–56, Santa Clara, CA. USENIX Association.
Publicado
03/11/2026
Como Citar
CATTAI, Luísa Leona; BALDASSIN, Alexandro; DANTAS, Allberson.
Avaliando a Estratificação de Memória na Inferência de LLMs somente em CPU. In: SIMPÓSIO EM SISTEMAS COMPUTACIONAIS DE ALTO DESEMPENHO (SSCAD), 27. , 2026, Natal/RN.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 70-84.
DOI: https://doi.org/10.5753/sscad.2026.34365.
