Gradient Estimation in Model-Based Reinforcement Learning: A Study on Linear Quadratic Environments

Resumo


Stochastic Value Gradient (SVG) methods underlie many recent achievements of model-based Reinforcement Learning agents in continuous state-action spaces. Despite their practical significance, many algorithm design choices still lack rigorous theoretical or empirical justification. In this work, we analyze one such design choice: the gradient estimator formula. We conduct our analysis on randomized Linear Quadratic Gaussian environments, allowing us to empirically assess gradient estimation quality relative to the actual SVG. Our results justify a widely used gradient estimator by showing it induces a favorable bias-variance tradeoff, which could explain the lower sample complexity of recent SVG methods.
Palavras-chave: Reinforcement learning, Model-based, Machine learning
Publicado
29/11/2021
LOVATTO, Ângelo Gregório; BUENO, Thiago Pereira; BARROS, Leliane Nunes de. Gradient Estimation in Model-Based Reinforcement Learning: A Study on Linear Quadratic Environments. In: BRAZILIAN CONFERENCE ON INTELLIGENT SYSTEMS (BRACIS), 10. , 2021, Online. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2021 . ISSN 2643-6264.