Integrated Design of Dam Size and Operations via Reinforcement Learning

In the water systems analysis literature and practice, planning (i.e., dam sizing) and management (i.e., operation design) have been for long time addressed as two weakly interconnected problems, and this often resulted in oversized, poorly performing infrastructures. Recently, several authors started exploring the interdependent nature of these two problems, introducing new integrated approaches to simultaneously design water infrastructures and their operations. Yet, the high computational burden is a likely downside of these methods, a large share of which require solving one optimal operation design problem for every candidate dam size, making it unfeasible to explore the entire planning and associated operation decision space. This paper contributes a novel reinforcement learning (RL)-based approach to integrate dam sizing and operation design while significantly containing computational costs with respect to alternative state-of-the-art methods. The approach first optimizes a single operating policy parametric in the dam size and then searches for the best reservoir size operated using this policy. The parametric policy is computed through a novel batch-mode RL algorithm, called Planning Fitted Q-Iteration (pFQI). The proposed RL approach is tested on a numerical case study, where the water infrastructure must be sized and operated to meet downstream users' water demand while minimizing construction costs. Results show that the proposed RL approach is able to identify more efficient system configurations with respect to traditional sizing approaches that neglect the optimal operation design phase. Furthermore, when compared with other integrated approaches, the pFQI algorithm is proven to be computationally more efficient.