BarySearch: Algorithm of Machine Learning Tuning Model with Barycenter Method

• Lucas Francisco Amaral Orosco Pellicer USP
• Felipe Miguel Pait USP

DOI: https://doi.org/10.5753/bresci.2020.11175

Abstract

In many Machine Learning applications, it is desirable to obtain the best set of hyperparameters to optimize the performance of the application. The problem of optimizing hyperparameters is known as tuning of Machine Learning models. Despite being an optimization problem, tuning faces complex difficulties, since the models are seen as black boxes without well-defined mathematical formulation. Also, there are problems with regions of oscillations and regions of vast plateaus. In this work, we present BarySearch, an algorithm that uses the equation of the barycenter without the need to calculate derivatives of the objective function. The BarySearch technique has shown promising results in practical tests of model tuning.

Keywords: Machine Learning, Optimization, Barycenter, Search Methods

References

Bergstra, J. and Bengio, Y. (2012). Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281–305.

Bergstra, J., Komer, B., Eliasmith, C., Yamins, D., and Cox, D. (2015). Hyperopt: A python library for model selection and hyperparameter optimization. Computational Science & Discovery, 8:014008.

Chollet, F. et al. (2015). Keras. ”https://keras.io”.

Dua, D. and Graff, C. (2017). UCI machine learning repository. "http://archive.ics.uci.edu/ml/index.php".

Eberhart, R. and Kennedy, J. S. (1995). A new optimizer using particle swarm theory. MHS’95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science, pages 39–43.

Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F. (2015). Efficient and robust automated machine learning. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems 28, pages 2962–2970. Curran Associates, Inc.

Hansen, N. and Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evol. Comput., 9(2):159–195.

Hutter, F., Hoos, H. H., and Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm configuration. In LION.

Jamieson, K. G. and Talwalkar, A. (2015). Non-stochastic best arm identification and hyperparameter optimization. In AISTATS.

Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. (1983). Optimization by simulated annealing. Science, 220 4598:671–80.

Koch, P., Golovidov, O., Gardner, S., Wujek, B., Griffin, J., and Xu, Y. (2018). Autotune: A derivative-free optimization framework for hyperparameter tuning. KDD, pages 443–452. ISBN: 978-1-4503-4887-4.

Le, T. T., Fu, W., and Moore, J. H. (2020). Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics, 36(1):250–256.

LeCun, Y., Kavukcuoglu, K., and Farabet, C. (2010). Convolutional networks and applications in vision. In Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pages 253–256. IEEE.

Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res., 18(1):6765–6816.

McKay, M. D. (1992). Latin hypercube sampling as a tool in uncertainty analysis of computer models. In Proceedings of the 24th Conference on Winter Simulation, WSC’92, pages 557–564, New York, NY, USA. ACM.

Pait, F. M. (2018). The Barycenter Method for Direct Optimization. ArXiv e-prints.

Rios, L. M. and Sahinidis, N. V. (2013). Derivative-free optimization: a review of algorithms and comparison of software implementations. Journal of Global Optimization, 56(3):1247–1293.

Snoek, J., Larochelle, H., and Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems 25, pages 2951–2959. Curran Associates, Inc.