Entropy-based prioritized sampling in Deep Q-learning

Online reinforcement agents take advantage of experience replay memory that allows them to reuse experiences from the past to re-learn, thus improving the overall efficiency of the learning process. Prioritizing on specific transitions during the sampling and replay increased the performance of learning even more, but in previous approaches the priority of the transitions was determined only by its TD error property. In this work, we introduce a novel criterion for evaluating the importance of the transition which is based on the Shannon's entropy of the agents perceived state space. Furthermore, we compare the performance of different criteria for prioritizing on one of the simulation environments included in REinforcejs framework. Experimental results show that DQ-ETD which uses a combination of entropy and TD error criterion outperforms the approaches based on the TD error criterion only such as DQ-TD.