Hyperparameter Analysis in Deep Learning Applications Using Provenance Data

• Débora B. Pina UFRJ
• Liliane Neves UFRJ
• Aline Paes UFRJ
• Daniel de Oliveira UFF
• Marta Mattoso UFRJ

DOI: https://doi.org/10.5753/sbbd.2019.8827

Abstract

Convolutional Neural Networks (CNN) training requires adjusting hyperparameters. Current solutions to help choosing the best hyperparameter configuration define their own representation to model the data derivation relationships. This proprietary representation makes data analysis and interoperability difficult. This paper proposes CNNProv, which adopts the W3C PROV standard to represent data derivation relationships to facilitate the analysis of hyperparameter configurations, thus contributing to the CNNs training phase. CNNProv captures provenance data and allows hyperparameter analysis at runtime. The experiments show the suitability of the W3C PROV for hyperparameter analysis, while contributing to the quality and reliability of CNN results, with negligible overhead of up to 4%.

Keywords: convolutional neural networks, hyperparameters, provenance, data analysis

References

Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467.

Bergstra, J. and Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281–305.

Freire, J., Koop, D., Santos, E., and Silva, C. T. (2008). Provenance for computational tasks: A survey. Computing in Science and Engineering, 10(3):11–21. DOI: https://doi.org/10.1109/mcse.2008.79

Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep learning. MIT press. DOI: https://doi.org/10.1007/s10710-017-9314-z

He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., and Han, S. (2018). Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the ECCV, pages 784–800. DOI: https://doi.org/10.1007/978-3-030-01234-2_48

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Adv. in neural inf. proc. sys., pages 1097–1105. DOI: https://doi.org/10.1145/3065386

Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324. DOI: https://doi.org/10.1109/5.726791

Moreau, L. and Groth, P. T. (2013). Provenance: An Introduction to PROV. Synthesis Lectures on the Semantic Web: Theory and Technology. Morgan & Claypool Publishers. DOI: https://doi.org/10.2200/s00528ed1v01y201308wbe007

Schelter, S., B¨ose, J.-H., Kirschnick, J., Klein, T., and Seufert, S. (2017). Automatically tracking metadata and provenance of machine learning experiments. In MLS workshop @ NIPS.

Silva, V., de Oliveira, D., Mattoso, M., and Valduriez, P. (2018). Dfanalyzer: Runtime dataflow analysis of scientific applications using provenance. PVLDB, 11(12):2082–2085. DOI: https://doi.org/10.14778/3229863.3236265

Souza, R., Neves, L., Azeredo, L., Luiz, R., Tady, E., Cavalin, P. R., and Mattoso, M. (2018). Towards a human-in-the-loop library for tracking hyperparameter tuning in deep learning development. In LADaS@VLDB.

Stamatogiannakis, M., Kazmi, H., Sharif, H., Vermeulen, R., Gehani, A., Bos, H., and Groth, P. (2016). Trade-offs in automatic provenance capture. In IPAW, pages 29–41. Springer. DOI: https://doi.org/10.1007/978-3-319-40593-3_3

Vanschoren, J., van Rijn, J. N., Bischl, B., and Torgo, L. (2014). Openml: Networked science in machine learning. SIGKDD Explor. Newsl., 15(2):49–60. DOI: https://doi.org/10.1145/2641190.2641198