Zeroth Order Policy Search Methods for Global Optimization Problems: An Experimental Study
Resumo
Os métodos Policy Search (PS) vem sendo utilizados nos últimos anos para se aprender, automaticamente, algoritmos de otimização, obtendo resultados animadores. Neste trabalho, consideramos métodos PS para aprender algoritmos de otimização para problemas de otimização global, considerando um cenário pouco estudado: funções de alta dimensionalidade e os algoritmos de otimização não possuem acesso às derivadas da função a ser otimizada. Os resultados apontam, que apesar das dificuldades, os algoritmos de otimização aprendidos têm um desempenho promissor no cenário estudado.
Referências
Andrychowicz, M., Denil, M., Colmenarejo, S. G., Hoffman, M. W., Pfau, D., Schaul, T., and de Freitas, N. (2016). Learning to learn by gradient descent by gradient descent. CoRR, abs/1606.04474.
Arulkumaran, K., Deisenroth, M. P., Brundage, M., and Bharath, A. A. (2017). Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym.
Chen, Y., Hoffman, M. W., Colmenarejo, S. G., Denil, M., Lillicrap, T. P., Botvinick, M., and de Freitas, N. (2017). Learning to learn without gradient descent by gradient descent.
Faury, L. and Vasile, F. (2018). Rover descent: Learning to optimize by learning to navigate on prototypical loss surfaces. CoRR, abs/1801.07222.
François-Lavet, V., Henderson, P., Islam, R., Bellemare, M. G., and Pineau, J. (2018). An Introduction to Deep Reinforcement Learning. Now Foundations and Trends.
Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In Proceedings of the 35th ICML 2018, volume 80, pages 1582–1591.
Golovin, D., Karro, J., Kochanski, G., Lee, C., Song, X., and Zhang, Q. (2019). Gradientless descent: High-dimensional zeroth-order optimization. CoRR, abs/1911.06317.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic algorithms and applications. CoRR, abs/1812.05905.
Li, K. and Malik, J. (2016). Learning to optimize. CoRR, abs/1606.01885.
Li, K. and Malik, J. (2017). Learning to optimize neural nets. CoRR, abs/1703.00441.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. CoRR, abs/1707.06347.
Snyman, J. A. and Wilke, D. N. (2018). Practical Mathematical Optimization. Springer International Publishing.
Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction (Second Edition). MIT Press.
Williams, R. J. and Peng, J. (1991). Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268.
Zhang, H., Sun, J., and Xu, Z. (2020). Learning to be global optimizer. CoRR, abs/2003.04521.
Zhang, H. and Yu, T. (2020). Taxonomy of Reinforcement Learning Algorithms, pages 125–133. Springer Singapore, Singapore.
Arulkumaran, K., Deisenroth, M. P., Brundage, M., and Bharath, A. A. (2017). Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym.
Chen, Y., Hoffman, M. W., Colmenarejo, S. G., Denil, M., Lillicrap, T. P., Botvinick, M., and de Freitas, N. (2017). Learning to learn without gradient descent by gradient descent.
Faury, L. and Vasile, F. (2018). Rover descent: Learning to optimize by learning to navigate on prototypical loss surfaces. CoRR, abs/1801.07222.
François-Lavet, V., Henderson, P., Islam, R., Bellemare, M. G., and Pineau, J. (2018). An Introduction to Deep Reinforcement Learning. Now Foundations and Trends.
Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In Proceedings of the 35th ICML 2018, volume 80, pages 1582–1591.
Golovin, D., Karro, J., Kochanski, G., Lee, C., Song, X., and Zhang, Q. (2019). Gradientless descent: High-dimensional zeroth-order optimization. CoRR, abs/1911.06317.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic algorithms and applications. CoRR, abs/1812.05905.
Li, K. and Malik, J. (2016). Learning to optimize. CoRR, abs/1606.01885.
Li, K. and Malik, J. (2017). Learning to optimize neural nets. CoRR, abs/1703.00441.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. CoRR, abs/1707.06347.
Snyman, J. A. and Wilke, D. N. (2018). Practical Mathematical Optimization. Springer International Publishing.
Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction (Second Edition). MIT Press.
Williams, R. J. and Peng, J. (1991). Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268.
Zhang, H., Sun, J., and Xu, Z. (2020). Learning to be global optimizer. CoRR, abs/2003.04521.
Zhang, H. and Yu, T. (2020). Taxonomy of Reinforcement Learning Algorithms, pages 125–133. Springer Singapore, Singapore.
Publicado
29/11/2021
Como Citar
SILVA FILHO, Moésio W. da; BARBOSA, Gabriel A.; MIRANDA, Péricles B. C.; NASCIMENTO, André C. A.; MELLO, Rafael Ferreira.
Zeroth Order Policy Search Methods for Global Optimization Problems: An Experimental Study. In: ENCONTRO NACIONAL DE INTELIGÊNCIA ARTIFICIAL E COMPUTACIONAL (ENIAC), 18. , 2021, Evento Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2021
.
p. 209-220.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2021.18254.
