Skip to content
← All publications

EpiCare: A reinforcement learning benchmark for dynamic treatment regimes

M. Hargrave, A. Spaeth, L. Grosenick (2024).

Advances in Neural Information Processing Systems (NeurIPS)

Paper Code PDF ★ Top-3 AI/ML venue

In brief

EpiCare is a benchmark for testing reinforcement learning on longitudinal treatment decisions. It finds that current offline RL methods and standard evaluation techniques struggle in the low-data settings typical of healthcare.

Abstract

Healthcare applications pose significant challenges to existing reinforcement learning (RL) methods due to implementation risks, limited data availability, short treatment episodes, sparse rewards, partial observations, and heterogeneous treatment effects. Despite significant interest in using RL to generate dynamic treatment regimes for longitudinal patient care scenarios, no standardized benchmark has yet been developed. To fill this need we introduce Episodes of Care (EpiCare), a benchmark designed to mimic the challenges associated with applying RL to longitudinal healthcare settings. We leverage this benchmark to test five state-of-the-art offline RL models as well as five common off-policy evaluation (OPE) techniques. Our results suggest that while offline RL may be capable of improving upon existing standards of care given sufficient data, its applicability does not appear to extend to the moderate to low data regimes typical of current healthcare settings. Additionally, we demonstrate that several OPE techniques standard in the medical RL literature fail to perform adequately on our benchmark. These results suggest that the performance of RL models in dynamic treatment regimes may be difficult to meaningfully evaluate using current OPE methods, indicating that RL for this application domain may still be in its early stages. We hope that these results along with the benchmark will facilitate better comparison of existing methods and inspire further research into techniques that increase the practical applicability of medical RL.

Cite

M. Hargrave, A. Spaeth, L. Grosenick (2024). EpiCare: A reinforcement learning benchmark for dynamic treatment regimes. Advances in Neural Information Processing Systems (NeurIPS).