BarySearch: Algoritmo de Tuning de Modelos de Machine Learning com o Metodo do Baricentro
Resumo
Em muitas aplicações de Machine Learning, é desejável obter o melhor conjunto de hiperparametros para otimizar o desempenho da aplicação. O problema de otimizar os hiperparametros é conhecido como tuning de modelos Machine Learning. Apesar de ser um problema de otimização, o tuning enfrenta dificuldades complexas, já que os modelos são vistos como caixas pretas sem formulação matemática bem definida. Além disso, há problemas com regioes de oscilações e regiões de grandes platôs. Nesse trabalho, nós apresentamos o BarySearch, um algoritmo que se utiliza da equação do baricentro sem necessidade de calcular derivadas da função objetivo. A técnica BarySearch demonstrou ter resultados promissores em testes praticos de tuning de modelos.
Palavras-chave:
Aprendizado de Máquina, Otimização, Baricentro, Métodos de Busca
Referências
Bergstra, J. and Bengio, Y. (2012). Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281–305.
Bergstra, J., Komer, B., Eliasmith, C., Yamins, D., and Cox, D. (2015). Hyperopt: A python library for model selection and hyperparameter optimization. Computational Science & Discovery, 8:014008.
Chollet, F. et al. (2015). Keras. ”https://keras.io”.
Dua, D. and Graff, C. (2017). UCI machine learning repository. "http://archive.ics.uci.edu/ml/index.php".
Eberhart, R. and Kennedy, J. S. (1995). A new optimizer using particle swarm theory. MHS’95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science, pages 39–43.
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F. (2015). Efficient and robust automated machine learning. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems 28, pages 2962–2970. Curran Associates, Inc.
Hansen, N. and Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evol. Comput., 9(2):159–195.
Hutter, F., Hoos, H. H., and Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm configuration. In LION.
Jamieson, K. G. and Talwalkar, A. (2015). Non-stochastic best arm identification and hyperparameter optimization. In AISTATS.
Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. (1983). Optimization by simulated annealing. Science, 220 4598:671–80.
Koch, P., Golovidov, O., Gardner, S., Wujek, B., Griffin, J., and Xu, Y. (2018). Autotune: A derivative-free optimization framework for hyperparameter tuning. KDD, pages 443–452. ISBN: 978-1-4503-4887-4.
Le, T. T., Fu, W., and Moore, J. H. (2020). Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics, 36(1):250–256.
LeCun, Y., Kavukcuoglu, K., and Farabet, C. (2010). Convolutional networks and applications in vision. In Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pages 253–256. IEEE.
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res., 18(1):6765–6816.
McKay, M. D. (1992). Latin hypercube sampling as a tool in uncertainty analysis of computer models. In Proceedings of the 24th Conference on Winter Simulation, WSC’92, pages 557–564, New York, NY, USA. ACM.
Pait, F. M. (2018). The Barycenter Method for Direct Optimization. ArXiv e-prints.
Rios, L. M. and Sahinidis, N. V. (2013). Derivative-free optimization: a review of algorithms and comparison of software implementations. Journal of Global Optimization, 56(3):1247–1293.
Snoek, J., Larochelle, H., and Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems 25, pages 2951–2959. Curran Associates, Inc.
Bergstra, J., Komer, B., Eliasmith, C., Yamins, D., and Cox, D. (2015). Hyperopt: A python library for model selection and hyperparameter optimization. Computational Science & Discovery, 8:014008.
Chollet, F. et al. (2015). Keras. ”https://keras.io”.
Dua, D. and Graff, C. (2017). UCI machine learning repository. "http://archive.ics.uci.edu/ml/index.php".
Eberhart, R. and Kennedy, J. S. (1995). A new optimizer using particle swarm theory. MHS’95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science, pages 39–43.
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F. (2015). Efficient and robust automated machine learning. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems 28, pages 2962–2970. Curran Associates, Inc.
Hansen, N. and Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evol. Comput., 9(2):159–195.
Hutter, F., Hoos, H. H., and Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm configuration. In LION.
Jamieson, K. G. and Talwalkar, A. (2015). Non-stochastic best arm identification and hyperparameter optimization. In AISTATS.
Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. (1983). Optimization by simulated annealing. Science, 220 4598:671–80.
Koch, P., Golovidov, O., Gardner, S., Wujek, B., Griffin, J., and Xu, Y. (2018). Autotune: A derivative-free optimization framework for hyperparameter tuning. KDD, pages 443–452. ISBN: 978-1-4503-4887-4.
Le, T. T., Fu, W., and Moore, J. H. (2020). Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics, 36(1):250–256.
LeCun, Y., Kavukcuoglu, K., and Farabet, C. (2010). Convolutional networks and applications in vision. In Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pages 253–256. IEEE.
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res., 18(1):6765–6816.
McKay, M. D. (1992). Latin hypercube sampling as a tool in uncertainty analysis of computer models. In Proceedings of the 24th Conference on Winter Simulation, WSC’92, pages 557–564, New York, NY, USA. ACM.
Pait, F. M. (2018). The Barycenter Method for Direct Optimization. ArXiv e-prints.
Rios, L. M. and Sahinidis, N. V. (2013). Derivative-free optimization: a review of algorithms and comparison of software implementations. Journal of Global Optimization, 56(3):1247–1293.
Snoek, J., Larochelle, H., and Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems 25, pages 2951–2959. Curran Associates, Inc.
Publicado
30/06/2020
Como Citar
PELLICER, Lucas Francisco Amaral Orosco; PAIT, Felipe Miguel.
BarySearch: Algoritmo de Tuning de Modelos de Machine Learning com o Metodo do Baricentro. In: BRAZILIAN E-SCIENCE WORKSHOP (BRESCI), 14. , 2020, Cuiabá.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2020
.
p. 1-8.
ISSN 2763-8774.
DOI: https://doi.org/10.5753/bresci.2020.11175.
