Gradient Estimation in Model-Based Reinforcement Learning: A Study on Linear Quadratic Environments

• Ângelo Gregório Lovatto USP https://orcid.org/0000-0002-0586-5412
• Thiago Pereira Bueno USP https://orcid.org/0000-0003-1999-9697
• Leliane Nunes de Barros USP https://orcid.org/0000-0002-9128-0188

Resumo

Stochastic Value Gradient (SVG) methods underlie many recent achievements of model-based Reinforcement Learning agents in continuous state-action spaces. Despite their practical significance, many algorithm design choices still lack rigorous theoretical or empirical justification. In this work, we analyze one such design choice: the gradient estimator formula. We conduct our analysis on randomized Linear Quadratic Gaussian environments, allowing us to empirically assess gradient estimation quality relative to the actual SVG. Our results justify a widely used gradient estimator by showing it induces a favorable bias-variance tradeoff, which could explain the lower sample complexity of recent SVG methods.

Palavras-chave: Reinforcement learning, Model-based, Machine learning