Constraint-Aware Deep Reinforcement Learning for vRAN Dynamic Placement

• Gabriel M. Almeida UFG
• Mohammad J. Abdel-Rahman Princess Sumaya University for Technology / Virginia Tech
• Kleber V. Cardoso UFG

DOI: https://doi.org/10.5753/sbrc.2024.1379

Resumo

The disaggregated and virtualized Radio Access Network (vRAN) is already present in 5G but tends to have increased adoption in 6G, mainly in the context of the Open RAN (O-RAN). Despite the potential benefits, the effective success of disaggregated vRAN depends on the efficient placement of Virtualized Network Functions (VNFs), which is influenced by the demand in the Radio Units (RUs). Several factors create dynamicity in this demand, but the number of users served by each RU is one of the most impacting. This problem has already been tackled in the recent literature, however, the works oversimplify important aspects that make their proposals inappropriate for real-world adoption. In this paper, we introduce a Deep Reinforcement Learning (DRL) agent that is constraint-aware, ensuring the solutions’ feasibility. We compare our DRL solution with existing optimization models and evaluate it under different scenarios, including the presence of Mobile Edge Computing (MEC) applications that compete for computing resources. Our contributions include a novel formulation, the implementation of a publicly available DRL agent, and insights into practical application scenarios for disaggregated vRAN optimization.

Referências

3rd Generation Partnership Project (3GPP) (2018). System Architecture for the 5G (Release 15). Technical report, 3GPP.

Abdel-Rahman, M., MAZIED, E., HASSAN, F., Teague, K., AL-SHAGGAH, A., MACKENZIE, A., Midkiff, S., and Cardoso, K. V. (2023). A stochastic optimization framework for joint RAN intelligent controller placement and RAN nodes assignment in O-RAN networks. TechRxiv. Preprint. DOI: 10.36227/techrxiv.24212046.v1.

Abdel-Rahman, M. J., Mazied, E. A., Teague, K., MacKenzie, A. B., and Midkiff, S. F. (2017). Robust controller placement and assignment in software-defined cellular networks. In Proc. IEEE Int. Conf. Comp. Commun Netw (ICCCN), pages 1–9.

Alba, A. and Kellerer, W. (2022). Dynamic functional split adaptation in next-generation radio access networks. IEEE Transactions on Network and Service Management, 19(3):3239–3263.

Almeida, G. et al. (2022a). Deep reinforcement learning for joint functional split and network function placement in vRAN. In Proc. IEEE Glob. Commun. Conf. (GLOBE-COM), pages 1229–1234.

Almeida, G. M. et al. (2023). A genetic algorithm for efficiently solving the virtualized radio access network placement problem. In Proc. IEEE Int. Conf. Commun. (ICC), pages 1874–1879.

Almeida, G. M., Pinto, L. d. L., Both, C. B., and Cardoso, K. V. (2022b). Optimal Joint Functional Split and Network Function Placement in Virtualized RAN With Splittable Flows. IEEE Wirel. Commun. Lett., 11:1684–88.

Dósa, G. (2007). The tight bound of first fit decreasing bin-packing algorithm. In Proc. Int. Symp. on Combin. Algor. Prob. and Exper Method., pages 1–11. Springer.

ETSI (2019). TS 138 401 – 5G; NG-RAN; Architecture description (3GPP TS 38.401 version 15.5.0 Release 15).

Fonseca, F., Correa, S., and Cardoso, K. (2019). Optimizing Allocation and Positioning in a Disaggregated Radio Access Network Aware of Paths Through the Core Infrastructure. In Proc. Simp. Bras. Red. Comp. e Sis. Dist. (SBRC), pages 791–804.

Garcia-Saavedra, A., Costa-Perez, X., Leith, D. J., and Iosifidis, G. (2018a). FluidRAN: Optimized vRAN/MEC Orchestration. In Proc. IEEE Int. Conf. Comput. Commun. (INFOCOM), pages 2366–2374.

Garcia-Saavedra, A. et al. (2018b). WizHaul: On the Centralization Degree of Cloud RAN Next Generation Fronthaul. IEEE Trans Mob Comput, 17(10):2452–2466.

Gupta, H., Antony Franklin, A., Kumar, M., and Tamma, B. R. (2022). Traffic-Aware Dynamic Functional Split for 5G Cloud Radio Access Networks. In 2022 IEEE 8th International Conference on Network Softwarization (NetSoft), pages 297–301.

Joda, R. et al. (2022). Deep reinforcement learning-based joint user association and CU–DU placement in O-RAN. IEEE Trans. Netw. Service Manag., 19(4):4097–4110.

Laghrissi, A. and Taleb, T. (2018). A survey on the placement of virtual resources and virtual network functions. IEEE Commun. Surv. Tutor., 21(2):1409–1434.

Larsen, L. M. P. et al. (2019). A Survey of the Functional Splits Proposed for 5G Mobile Crosshaul Networks. IEEE Commun. Surv. Tutor., 21(1):146–172.

Lee, H., Jang, Y., Song, J., and Yeon, H. (2021). O-RAN AI/ML Workflow Implementation of Personalized Network Optimization via Reinforcement Learning. In Proc. IEEE Glob. Commun. Conf. (GLOBECOM), pages 1–6.

Liu, Q., Choi, N., and Han, T. (2021). Constraint-aware deep reinforcement learning for end-to-end resource orchestration in mobile networks. In Proc. IEEE Int. Conf. Netw. Prot. (ICNP), pages 1–11.

Morais, F. Z. et al. (2023). Placeran: Optimal placement of virtualized network functions in beyond 5g radio access networks. IEEE Trans. Mob. Comput., 22(9):5434–5448.

Murti, F., Ayala-Romero, J. A., Garcia-Saavedra, A., Costa-Pérez, X., and Iosifidis, G. (2021). An Optimal Deployment Framework for Multi-Cloud Virtualized Radio Access Networks. IEEE Trans. Wirel. Commun., 20(4):2251–2265.

Murti, F. W. et al. (2022). Constrained deep reinforcement based functional split optimization in virtualized rans. IEEE Trans. Wirel. Commun., 21(11):9850–64.

Murti, F. W. et al. (2023). Deep reinforcement learning for orchestrating cost-aware reconfigurations of vRANs. IEEE Trans. Netw. Service Manag.

Open RAN Alliance (2022). Cloud Architecture and Deployment Scenarios for O-RAN Virtualized RAN. Technical report, O-RAN Alliance. white paper.

Pamuklu, T. et al. (2021). Reinforcement learning based dynamic function splitting in disaggregated green open RANs. Proc. IEEE Int. Conf. Comm. (ICC), pages 1–6.