Discovering Subgoals from Trajectories Using Empowerment
Resumo
Reinforcement learning algorithms struggle in complex environments with sparse rewards due to inefficient exploration. Identifying subgoals to guide exploration and structure learning is a promise and a challenge. This work presents a novel method to discover subgoals from trajectories by leveraging empowerment, an information-theoretic measure of an agent’s potential to influence its environment. The main hypothesis is that states with high empowerment correspond to strategic locations, such as bottlenecks, which serve as subgoals. We evaluate our method in grid-world navigation tasks, demonstrating that it successfully identifies important states without requiring reward signals or successful trajectories, and improves agent performance.Referências
Cover, T. M. (1999). Elements of information theory. John Wiley & Sons.
Klyubin, A. S., Polani, D., and Nehaniv, C. L. (2005). All else being equal be empowered. In European Conference on Artificial Life, pages 744–753. Springer.
McGovern, A. and Barto, A. G. (2001). Automatic discovery of subgoals in reinforcement learning using diverse density. In ICML, volume 1, pages 361–368.
Şimşek, Ö. and Barto, A. G. (2004). Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proc. of ICML, page 95.
Şimşek, Ö., Wolfe, A. P., and Barto, A. G. (2005). Identifying useful subgoals in rein-forcement learning by local graph partitioning. In Proceedings of the 22nd international conference on Machine learning, pages 816–823.
Sutton, R., Precup, D., and Singh, S. (1999). Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(12):181–211.
Sutton, R. S. (2018). Reinforcement learning: An introduction. A Bradford Book.
Sutton, R. S., Machado, M. C., Holland, G. Z., Szepesvari, D., Timbers, F., Tanner, B., and White, A. (2023). Reward-respecting subtasks for model-based reinforcement learning. Artificial Intelligence, 324:104001.
Klyubin, A. S., Polani, D., and Nehaniv, C. L. (2005). All else being equal be empowered. In European Conference on Artificial Life, pages 744–753. Springer.
McGovern, A. and Barto, A. G. (2001). Automatic discovery of subgoals in reinforcement learning using diverse density. In ICML, volume 1, pages 361–368.
Şimşek, Ö. and Barto, A. G. (2004). Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proc. of ICML, page 95.
Şimşek, Ö., Wolfe, A. P., and Barto, A. G. (2005). Identifying useful subgoals in rein-forcement learning by local graph partitioning. In Proceedings of the 22nd international conference on Machine learning, pages 816–823.
Sutton, R., Precup, D., and Singh, S. (1999). Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(12):181–211.
Sutton, R. S. (2018). Reinforcement learning: An introduction. A Bradford Book.
Sutton, R. S., Machado, M. C., Holland, G. Z., Szepesvari, D., Timbers, F., Tanner, B., and White, A. (2023). Reward-respecting subtasks for model-based reinforcement learning. Artificial Intelligence, 324:104001.
Publicado
12/11/2025
Como Citar
THOMASINI, Luiz A.; SOUZA, Arturo de; RAMOS, Gabriel de Oliveira.
Discovering Subgoals from Trajectories Using Empowerment. In: ESCOLA REGIONAL DE APRENDIZADO DE MÁQUINA E INTELIGÊNCIA ARTIFICIAL DA REGIÃO SUL (ERAMIA-RS), 1. , 2025, Porto Alegre/RS.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 412-415.
DOI: https://doi.org/10.5753/eramiars.2025.16775.
