Continuous Voltage Regulation in Active Distribution Networks Using Twin Delayed Deep Deterministic Policy Gradient

Abstract

Active Distribution Networks (ADNs), characterized by high levels of penetration by Electric Vehicles (EVs) and renewable energy sources (RES), lead to a high degree of uncertainty and control challenges for many system operators. Most previous studies examined methods for controlling voltage, but the Finite Action Space (AS) set a limit on the control granularity and scalability of the methods investigated. With this in mind, the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is presented, a Continuous-Action (CA) actor-critic reinforcement learning algorithm in which the same system model, reward structure, and EV participant constraints are kept for a direct comparison. The TD3-based controller provided real-valued control actions for distribution system resources, fully approving of the additional flexibility provided by continuous ASs. The issue of value overestimation was overcome by combining TD3 with twin Critic Networks (CNs), while the features of target smoothing and delayed policy updates are also introduced to strengthen the stability and convergence of the learning algorithm. The simulations with the IEEE 33-bus (IEEE-33) and IEEE 123-bus (IEEE-123) systems highlighted improvements in voltage control granularity, convergence speed, and scalability across uncertain State Spaces (SSs).