Reinforcement Learning for Dynamic Pump Scheduling under Demand Uncertainty

Reliable and cost-efficient scheduling of pumps is an important task in the daily operations of urban water distribution networks (WDNs). In this work, we address the scheduling of variable-speed pumps using reinforcement learning (RL), which allows network controls to adapt to changes in demand in real-time after a data-driven training phase. Previous contributions have shown the general suitability of RL for control tasks in WDNs [1], [2]. However, most of them assume deterministically known demand patterns (cf. [1]) or consider uncertainty only for valve control (cf. [2]). As RL algorithms can handle uncertain environments, we explore their potential for dynamic scheduling of the network’s pumps under uncertain demand patterns. Our optimisation goal is to train a policy that complies with upper and lower pressure bounds at all nodes in the network while minimising the cost of pumping. To this end, we make use of the Soft Actor-Critic algorithm (SAC) [3]. Data for training and testing is collected using the EPANET simulator for two benchmark networks (Net1 and Anytown) with uncertainties applied to various network parameters. In all setups, the controller is trained without nodal demand information. Our study shows promising results for a pump scheduler that can reduce energy cost by a significant amount while complying with pressure bounds even for unseen scenarios.