Wide-Area Feedback Control for Renewables-Heavy Power Systems: A Comparative Study of Reinforcement Learning and Lyapunov-Based Design

Abstract: As renewable energy sources become more prevalent, accurately modeling power grid dynamics is becoming increasingly more complex. Concurrently, data acquisition and realtime system state monitoring are becoming more available for control centers. This motivates shifting from \textit{model- and Lyapunov-based} feedback controller designs toward \textit{model-free} ones. Reinforcement learning (RL) has emerged as a key tool for designing model-free controllers. Various studies have been carried out to study voltage/frequency control strategies via RL. However, usually a simplified system model is used neglecting detailed dynamics of solar, wind, and composite loads -- and damping system-wide oscillations and modeling power flows are all usually ignored. To that end, we pose an optimal feedback control problem for a detailed renewables-heavy power system, defined by a set of nonlinear differential algebraic equations (NDAE). The control problem is solved using a completely model-free design via RL as well as using a model-based approach built upon the Lyapunov stability theory with guarantees. The paper in its essence seeks to explore whether data-driven feedback control should be used in power grids over its model-driven counterpart. Theoretical developments and thorough case studies are presented with an eye on this exploration. Finally, a detailed analysis is provided to delineate the strengths and weaknesses of both approaches for renewables-heavy grids.