TY - JOUR
T1 - Inverse reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units
AU - Yu, Chao
AU - LIU, Jiming
AU - Zhao, Hongyi
N1 - Funding Information:
This work is supported by the Hongkong Scholar Program under Grant No. XJ2017028, and Dalian High Level Talent Innovation Support Program under Grant 2017RQ008.
PY - 2019/4/9
Y1 - 2019/4/9
N2 - Background: Reinforcement learning (RL) provides a promising technique to solve complex sequential decision making problems in health care domains. To ensure such applications, an explicit reward function encoding domain knowledge should be specified beforehand to indicate the goal of tasks. However, there is usually no explicit information regarding the reward function in medical records. It is then necessary to consider an approach whereby the reward function can be learned from a set of presumably optimal treatment trajectories using retrospective real medical data. This paper applies inverse RL in inferring the reward functions that clinicians have in mind during their decisions on weaning of mechanical ventilation and sedative dosing in Intensive Care Units (ICUs). Methods: We model the decision making problem as a Markov Decision Process, and use a batch RL method, Fitted Q Iterations with Gradient Boosting Decision Tree, to learn a suitable ventilator weaning policy from real trajectories in retrospective ICU data. A Bayesian inverse RL method is then applied to infer the latent reward functions in terms of weights in trading off various aspects of evaluation criterion. We then evaluate how the policy learned using the Bayesian inverse RL method matches the policy given by clinicians, as compared to other policies learned with fixed reward functions. Results: Results show that the inverse RL method is capable of extracting meaningful indicators for recommending extubation readiness and sedative dosage, indicating that clinicians pay more attention to patients' physiological stability (e.g., heart rate and respiration rate), rather than oxygenation criteria (FiO 2, PEEP and SpO 2) which is supported by previous RL methods. Moreover, by discovering the optimal weights, new effective treatment protocols can be suggested. Conclusions: Inverse RL is an effective approach to discovering clinicians' underlying reward functions for designing better treatment protocols in the ventilation weaning and sedative dosing in future ICUs.
AB - Background: Reinforcement learning (RL) provides a promising technique to solve complex sequential decision making problems in health care domains. To ensure such applications, an explicit reward function encoding domain knowledge should be specified beforehand to indicate the goal of tasks. However, there is usually no explicit information regarding the reward function in medical records. It is then necessary to consider an approach whereby the reward function can be learned from a set of presumably optimal treatment trajectories using retrospective real medical data. This paper applies inverse RL in inferring the reward functions that clinicians have in mind during their decisions on weaning of mechanical ventilation and sedative dosing in Intensive Care Units (ICUs). Methods: We model the decision making problem as a Markov Decision Process, and use a batch RL method, Fitted Q Iterations with Gradient Boosting Decision Tree, to learn a suitable ventilator weaning policy from real trajectories in retrospective ICU data. A Bayesian inverse RL method is then applied to infer the latent reward functions in terms of weights in trading off various aspects of evaluation criterion. We then evaluate how the policy learned using the Bayesian inverse RL method matches the policy given by clinicians, as compared to other policies learned with fixed reward functions. Results: Results show that the inverse RL method is capable of extracting meaningful indicators for recommending extubation readiness and sedative dosage, indicating that clinicians pay more attention to patients' physiological stability (e.g., heart rate and respiration rate), rather than oxygenation criteria (FiO 2, PEEP and SpO 2) which is supported by previous RL methods. Moreover, by discovering the optimal weights, new effective treatment protocols can be suggested. Conclusions: Inverse RL is an effective approach to discovering clinicians' underlying reward functions for designing better treatment protocols in the ventilation weaning and sedative dosing in future ICUs.
KW - Intensive care units
KW - Inverse learning
KW - Mechanical ventilation
KW - Reinforcement learning
KW - Sedative dosing
UR - http://www.scopus.com/inward/record.url?scp=85064130719&partnerID=8YFLogxK
U2 - 10.1186/s12911-019-0763-6
DO - 10.1186/s12911-019-0763-6
M3 - Journal article
C2 - 30961594
AN - SCOPUS:85064130719
SN - 1472-6947
VL - 19
JO - BMC Medical Informatics and Decision Making
JF - BMC Medical Informatics and Decision Making
M1 - 57
ER -