Risk Sensitive Policies for the Control of the Spread of Infectious Diseases
Abstract
The definition of policies to control the spread of infectious diseases, such as COVID19, is an important issue for society and for the government agents, responsable for the controlling measures. The majority of the vast number of recent research on this topic, uses past data to estimate the disease behavior in the future, considering an arbitrary policy. However, the use of automated planning techniques based on Markov Decision Processes (MDPs) has been shown to be more efficient to compute optimized control policies. One of the most common model used to control infectious diseases is the "SIR with vaccination model" in which the dynamics of the susceptible, infected and recovered individuals amount is controled by vaccination policies. But since this is a problem of saving lives, it is also necessary to take into account the attitude towards risk of a decision maker. Thus, the purpose of this work is to use risk-sensitive MDPs to find optimized policies to control the spread of the considered infectious diseases, taking into account the parameters of the SIR with risk model. The results show that vaccination policies depend on both the basic reproductive rate $R_0$, as usual, and the attitude towards the risk of the government agents.
References
Denardo, E. V. and Rothblum, U. G. (1979). Optimal stopping, exponential utility, and linear programming. Mathematical Programming, 16(1):228–244.
Diekmann, O. and Heesterbeek, J. (2000). Mathematical Epidemiology of Infectious Diseases: model building, analysis and interpretation. John Wiley & Son.
Filar, J. A., Kallenberg, L. C. M., and Lee, H.-M. (1989). Variance-penalized Markov decision processes. Mathematics of Operations Research, 14(1):147–161.
Filar, J. A., Krass, D., Ross, K. W., and Ross, K. W. (1995). Percentile performance criteria for limiting average Markov decision processes. IEEE Transactions on Automatic Control, 40(1):2–10.
Garcı́a, J. and Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res., 16(1):1437–1480.
Heesterbeek, J. and Dietz, K. (1996). The concept of ro in epidemic theory. Statistica Neerlandica, 50(1):89–110.
Heesterbeek, J. A. P. (2002). A brief history of r0 and a recipe for its calculation. Acta biotheoretica, 50(3):189–204.
Heffernan, J. M., Smith, R. J., and Wahl, L. M. (2005). Perspectives on the basic reproductive ratio. Journal of The Royal Society Interface, 2:281 – 293.
Hou, P., Yeoh, W., and Varakantham, P. (2014). Revisiting risk-sensitive MDPs: New algorithms and results. In Proceedings of the Twenty-Fourth International Conference on Automated Planning and Scheduling, ICAPS 2014, Portsmouth, New Hampshire, USA, June 21-26, 2014.
Hou, P., Yeoh, W., and Varakantham, P. (2016). Solving risk-sensitive POMDPs with and without cost observations. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., pages 3138– 3144.
Howard, R. A. and Matheson, J. E. (1972). Risk-sensitive Markov decision processes. Management science, 18(7):356–369.
Jaquette, S. C. (1976). A utility criterion for Markov decision processes. Management Science, 23(1):43–49.
Keeling, M. J. and Rohani, P. (2011). Modeling infectious diseases in humans and animals. Princeton University Press.
Kermack, W. O. and McKendrick, A. G. (1927). A contribution to the mathematical theory of epidemics. Proceedings of the Royal Society of London A: mathematical, physical and engineering sciences, 115(772):700–721.
Kinathil, S., Soh, H., and Sanner, S. (2017). Analytic decision analysis via symbolic dynamic programming for parameterized hybrid MDPs. In Proceedings of the 27th Conference on Automated Planning and Scheduling (ICAPS-17), Pittsburgh, PA.
Lefévre, C. (1981). Optimal control of a birth and death epidemic process. Operations Research, 29(5):971–982.
Mihatsch, O. and Neuneier, R. (2002). Risk-sensitive reinforcement learning. Machine Learning, 49(2):267–290.
Nasir, A. and Rehman, H. (2017). Optimal control for stochastic model of epidemic infections. In 2017 14th International Bhurban Conference on Applied Sciences and Technology (IBCAST), pages 278–284. IEEE.
Patek, S. D. (2001). On terminating Markov decision processes with a risk-averse objective function. Automatica, 37(9):1379–1386.
Puterman, M. L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., New York, NY, USA, 1st edition.
Rothblum, U. G. (1984). Multiplicative Markov decision chains. Mathematics of Operations Research, 9(1):6–24.
Sobel, M. J. (1982). The variance of discounted Markov decision processes. Journal of Applied Probability, 19(4):794–802.
Yaesoubi, R. and Cohen, T. (2011a). Dynamic health policies for controlling the spread of emerging infections: influenza as an example. PloS one, 6(9):e24043.
Yaesoubi, R. and Cohen, T. (2011b). Generalized Markov models of infectious disease spread: A novel framework for developing dynamic health policies. European Journal of Operational Research, 215(3):679–687.
Yaesoubi, R. and Cohen, T. (2016). Identifying cost-effective dynamic policies to control epidemics. Statistics in medicine, 35(28):5189–5209.
Yu, S. X., Lin, Y., and Yan, P. (1998). Optimization models for the first arrival target distribution function in discrete time. Journal of Mathematical Analysis and Applications, 225(1):193 – 223.
