Abstract
The great complexity and size of electrical power systems makes their operation and control a challenging task. Maintaining a stable voltage profile to assure the security and stability of the system is one of many tasks that must be conducted daily by power system operators and its automatic control equipment. This work proposes a deep reinforcement learning framework for controlling the equipment responsible for keeping the voltages across the system buses within their limits. More specifically, a smart agent that is capable of deciding the best course of action in order to keep the system’s voltages within a specified range while taking into account system’s conditions is proposed. Besides the traditional deep reinforcement learning approach, three novel reinforcement learning variations named windowed, ensemble and windowed ensemble Q-Learning, which alter the agent’s learning process for voltage control, are presented and tested on IEEE 13, 37 and 123 bus systems, simulated on OpenDSS.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Baran, M.: Ming-Yung Hsu: Volt/VAr control at distribution substations. IEEE Trans. Power Syst. 14(1), 312–318 (1999). https://doi.org/10.1109/59.744549. https://ieeexplore.ieee.org/document/744549/
Borozan, V., Baran, M., Novosel, D.: Integrated volt/VAr control in distribution systems. In: 2001 IEEE Power Engineering Society Winter Meeting. Conference Proceedings (Cat. No. 01CH37194), Columbus, OH, USA, vol. 3, pp. 1485–1490. IEEE (2001). https://doi.org/10.1109/PESW.2001.917328. http://ieeexplore.ieee.org/document/917328/
Corsi, S.: Voltage Control and Protection in Electrical Power Systems: From System Components to Wide-Area Control. Springer, London (2015). https://doi.org/10.1007/978-1-4471-6636-8
Custódio, G., Ochoa, L., Trindade, F., Alpcan, T.: Using Q-learning for OLTC voltage regulation in PV-rich distribution networks (2020)
Diao, R., Wang, Z., Shi, D., Chang, Q., Duan, J., Zhang, X.: Autonomous voltage control for grid operation using deep reinforcement learning (2019). http://arxiv.org/abs/1904.10597
Franco, J.F., Rider, M.J., Lavorato, M., Romero, R.: A mixed-integer LP model for the optimal allocation of voltage regulators and capacitors in radial distribution systems. Int. J. Electr. Power Energy Syst. 48, 123–130 (2013). https://doi.org/10.1016/j.ijepes.2012.11.027. https://linkinghub.elsevier.com/retrieve/pii/S0142061512006801
Gallego, R., Monticelli, A., Romero, R.: Optimal capacitor placement in radial distribution networks. IEEE Trans. Power Syst. 16(4), 630–637 (2001). https://doi.org/10.1109/59.962407. https://ieeexplore.ieee.org/document/962407/
Gu, Z., Rizy, D.: Neural networks for combined control of capacitor banks and voltage regulators in distribution systems. IEEE Trans. Power Deliv. 11(4), 1921–1928 (1996). https://doi.org/10.1109/61.544277. https://ieeexplore.ieee.org/document/544277/
van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double Q-learning (2016). http://arxiv.org/abs/1509.06461
Hessel, M., et al.: Rainbow: combining improvements in deep reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
Homaee, O., Zakariazadeh, A., Jadid, S.: Real-time voltage control algorithm with switched capacitors in smart distribution system in presence of renewable generations. Int. J. Electr. Power Energy Syst. 54, 187–197 (2014). https://doi.org/10.1016/j.ijepes.2013.07.010. http://linkinghub.elsevier.com/retrieve/pii/S0142061513003074
Hu, Z., Wang, X., Chen, H., Taylor, G.: Volt/Var control in distribution systems using a time-interval based approach. IEE Proc.-Gener. Transm. Distrib. 150(5), 548 (2003). https://doi.org/10.1049/ip-gtd:20030562
Khanabadi, M., Ghasemi, H., Doostizadeh, M.: Optimal transmission switching considering voltage security and N-1 contingency analysis. IEEE Trans. Power Syst. 28(1), 542–550 (2013). https://doi.org/10.1109/TPWRS.2012.2207464. https://ieeexplore.ieee.org/document/6253283/
Levitin, G., Kalyuzhny, A., Shenkman, A., Chertkov, M.: Optimal capacitor allocation in distribution systems using a genetic algorithm and a fast energy loss computation technique. IEEE Trans. Power Deliv. 15(2), 623–628 (2000). https://doi.org/10.1109/61.852995. https://ieeexplore.ieee.org/document/852995/
Liang, R.H., Chen, Y.K., Chen, Y.T.: Volt/Var control in a distribution system by a fuzzy optimization approach. Int. J. Electr. Power Energy Syst. 33(2), 278–287 (2011). https://doi.org/10.1016/j.ijepes.2010.08.023. https://linkinghub.elsevier.com/retrieve/pii/S0142061510001596
Liu, Y., Zhang, P., Qiu, X.: Optimal volt/var control in distribution systems. Int. J. Electr. Power Energy Syst. 24(4), 271–276 (2002). https://doi.org/10.1016/S0142-0615(01)00032-1. https://linkinghub.elsevier.com/retrieve/pii/S0142061501000321
Meier, A.V.: Electric Power Systems: A Conceptual Introduction. Wiley survival guides in engineering and science, IEEE Press: Wiley-Interscience, Hoboken (2006). oCLC: ocm62616191
Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Niknam, T.: A new approach based on ant colony optimization for daily Volt/Var control in distribution networks considering distributed generators. Energy Convers. Manag. 49(12), 3417–3424 (2008). https://doi.org/10.1016/j.enconman.2008.08.015. https://linkinghub.elsevier.com/retrieve/pii/S0196890408003087
Niknam, T., Firouzi, B.B., Ostadi, A.: A new fuzzy adaptive particle swarm optimization for daily Volt/Var control in distribution networks considering distributed generators. Appl. Energy 87(6), 1919–1928 (2010). https://doi.org/10.1016/j.apenergy.2010.01.003. https://linkinghub.elsevier.com/retrieve/pii/S030626191000005X
Rashid, M.H. (ed.): Electric renewable energy systems. Elsevier/AP, Academic Press is an imprint of Elsevier, Amsterdam (2016)
Ribeiro, L.C., Schumann Minami, J.P.O., Bonatto, B.D., Ribeiro, P.F., de Souza, A.C.Z.: Voltage control simulations in distribution systems with high penetration of PVs using the OpenDSS. In: 2018 Simposio Brasileiro de Sistemas Eletricos (SBSE), pp. 1–6. IEEE. https://doi.org/10.1109/SBSE.2018.8395639. https://ieeexplore.ieee.org/document/8395639/
Roytelman, I., Wee, B., Lugtu, R.: Volt/var control algorithm for modern distribution management system. IEEE Trans. Power Syst. 10(3), 1454–1460 (1995). https://doi.org/10.1109/59.466504. https://ieeexplore.ieee.org/document/466504/
Saric, A.T., Stankovic, A.M.: A robust algorithm for Volt/Var control. In: 2009 IEEE/PES Power Systems Conference and Exposition, Seattle, WA, USA, pp. 1–8. IEEE (2009). https://doi.org/10.1109/PSCE.2009.4840211. https://ieeexplore.ieee.org/document/4840211/
Schneider, K.P., et al.: Analytic considerations and design basis for the IEEE distribution test feeders. IEEE Trans. Power Syst. 33(3), 3181–3188 (2018). https://doi.org/10.1109/TPWRS.2017.2760011. https://ieeexplore.ieee.org/document/8063903/
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning Series, 2nd edn. The MIT Press, Cambridge (2018)
Vlachogiannis, J., Hatziargyriou, N.: Reinforcement learning for reactive power control. IEEE Trans. Power Syst. 19(3), 1317–1325 (2004). https://doi.org/10.1109/TPWRS.2004.831259. https://ieeexplore.ieee.org/document/1318666/
Wood, A.J., Wollenberg, B.F., Sheblé, G.B.: Power Generation, Operation, and Control, 3rd edn. Hoboken, Wiley-IEEE (2013)
Xu, H., Domínguez-García, A.D., Sauer, P.W.: Optimal tap setting of voltage regulation transformers using batch reinforcement learning. IEEE Trans. Power Syst. 35(3), 1990–2001 (2019). https://doi.org/10.1109/TPWRS.2019.2948132. https://arxiv.org/abs/1807.10997
Yang, Q., Wang, G., Sadeghi, A., Giannakis, G.B., Sun, J.: Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning. arXiv:1904.09374 (2019)
Zhang, W., Liu, Y.: Multi-objective reactive power and voltage control based on fuzzy optimization strategy and fuzzy adaptive particle swarm. IEEE Trans. Power Syst. 30(9), 525–532 (2019). https://doi.org/10.1016/j.ijepes.2008.04.005. https://linkinghub.elsevier.com/retrieve/pii/S0142061508000380
Acknowledgments
The authors would like to thank Companhia de Transmissão de Energia Elétrica Paulista (ISA-CTEEP) for the financial support to Project PD-00068-0044-2019 - intelligent real-time decision support tool for transmission operations centers, developed under the Research and Development program of the National Electric Energy Agency (ANEEL R &D), which the engineering company carried out Radix Engenharia e Software S/A, Rio de Janeiro, Brazil.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Barg, M.W. et al. (2023). Deep Reinforcement Learning for Voltage Control in Power Systems. In: Naldi, M.C., Bianchi, R.A.C. (eds) Intelligent Systems. BRACIS 2023. Lecture Notes in Computer Science(), vol 14196. Springer, Cham. https://doi.org/10.1007/978-3-031-45389-2_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-45389-2_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45388-5
Online ISBN: 978-3-031-45389-2
eBook Packages: Computer ScienceComputer Science (R0)