Experience replay in reinforcement learning: A systematic review

Background: Experience Replay (ER) has become a cornerstone of modern reinforcement learning (RL), enabling improved sample efficiency and training stability by reusing past experiences. Since its introduction, a diverse set of ER methods has emerged, including prioritized, hindsight, reservoir-based, episodic, and compressed replay schemes. While several narrative surveys exist, a systematic, transparent, and reproducible synthesis of this literature is still lacking. Objective: This article provides a comprehensive systematic literature review (SLR) and taxonomy of ER methods in RL. We aim to (i) categorize existing approaches across key methodological dimensions, (ii) quantify publication trends and application domains, and (iii) identify research gaps and reproducibility issues. Methods: We followed PRISMA guidelines to conduct a structured search across major scientific databases (Scopus, IEEE Xplore, ACM Digital Library, and Springer). An initial pool of 3600 records was identified, from which 250 studies were assessed in full text after screening. Following quality assessment and snowballing, a final corpus of 200 studies was included in the systematic synthesis. Results: Our systematic analysis identifies clear methodological trends and clusters in experience replay research. Prioritized replay is the most widely adopted advanced sampling strategy, while hindsight replay predominates in goal-conditioned and sparse-reward tasks. The field also shows growing diversification toward episodic, memory-efficient, and multi-agent replay designs. However, reproducibility remains a concern, with only a minority of studies providing accessible code or complete experimental documentation. Conclusions: This review provides the first systematic, taxonomy-driven synthesis of ER methods in RL, highlighting both dominant trends and under-explored areas such as uncertainty-aware sampling and scalable memory architectures. All extracted datasets, tables, and screening information are released as open supplementary material to support future research.