Regularized Regression solution with Support Vectors through Linear Programming
Abstract
In this work two regression methods based on support vector theory were introduced. These methods aim to find sparse solutions and are solvable by linear programming. One of them is only applicable to linear regression however the other can be extended to the nonlinear case through kernel methods. The proposed methods obtained numerical results close to state of the art methods.
Keywords:
Support Vector Regression, SVR, Regularization, Variable Selection, Linear Programming, LP
References
Alves, A., Chaparro Pinzon, A., Costa, R., Silva, M., Vieira, E., Mendonça, I., Viana, V., and Lôbo, R. (2019). Multiple regression and machine learning based methods for cacass traits and saleable meat cuts prediction using non-invasive in vivo measurements in commercial lambs. Small Ruminant Research, 171:49–56. cited By 0.
Bi, J., Bennett, K., Embrechts, M., Breneman, C., and Song, M. (2003). Dimensionality reduction via sparse support vector machines. J. Mach. Learn. Res., 3:1229–1243.
Boser, B. E., Guyon, I. M., and Vapnik, V. N. (1992). A training algorithm for optimal margin classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, COLT ’92, pages 144–152, New York, NY, USA. ACM.
Boucher, T., Ozanne, M., Carmosino, M., Dyar, M., Mahadevan, S., Breves, E., Lepore, K., and Clegg, S. (2015). A study of machine learning regression methods for major elemental analysis of rocks using laser-induced breakdown spectroscopy. Spectrochimica Acta - Part B Atomic Spectroscopy, 107:1–10. cited By 30.
Chang, C.-C. and Lin, C.-J. (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27. Software available at http://www.csie.ntu.edu.tw/˜cjlin/libsvm.
De Rivas, B., Vivancos, J.-L., Ordieres-Meré, J., and Capuz-Rizo, S. (2017). Determnation of the total acid number (tan) of used mineral oils in aviation engines by ftir using regression models. Chemometrics and Intelligent Laboratory Systems, 160:32– 39. cited By 6.
Drucker, H., Burges, C. J. C., Kaufman, L., Smola, A. J., and Vapnik, V. (1997). Support vector regression machines. In Mozer, M. C., Jordan, M. I., and Petsche, T., editors, Advances in Neural Information Processing Systems 9, pages 155–161. MIT Press.
Dua, D. and Graff, C. (2017). UCI machine learning repository.
Friedman, J., Hastie, T., and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1):1–22.
Gelius-Dietrich, G. (2017). cplexAPI: R Interface to C API of IBM ILOG CPLEX. R package version 1.3.3.
Jaggi, M. (2013). An equivalence between the lasso and support vector machines. Regularization, optimization, kernels, and support vector machines, chap 1.
Mercer, J. (1909). Functions of positive and negative type, and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 209:415– 446.
Meyer, D., Dimitriadou, E., Hornik, K., Weingessel, A., and Leisch, F. (2017). e1071: Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien. R package version 1.6-8.
Smola, A., Scholkopf, B., and Ratsch, G. (1999). Linear programs for automatic accuracy control in regression. In 1999 Ninth International Conference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470), volume 2, pages 575–580 vol.2.
Smola, A. J. and Schölkopf, B. (2004). A tutorial on support vector regression. Statistics and Computing, 14(3):199–222.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1):267–288.
Tikhonov, A. N. (1963). On the solution of ill-posed problems and the method of regularization. Doklady Akademii Nauk SSSR, 151(3):501–504.
Zhou, Q., Chen, W., Song, S., Gardner, J. R., Weinberger, K. Q., and Chen, Y. (2015). A reduction of the elastic net to support vector machines with an application to gpu computing. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, pages 3210–3216. AAAI Press.
Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B, 67:301–320.
Bi, J., Bennett, K., Embrechts, M., Breneman, C., and Song, M. (2003). Dimensionality reduction via sparse support vector machines. J. Mach. Learn. Res., 3:1229–1243.
Boser, B. E., Guyon, I. M., and Vapnik, V. N. (1992). A training algorithm for optimal margin classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, COLT ’92, pages 144–152, New York, NY, USA. ACM.
Boucher, T., Ozanne, M., Carmosino, M., Dyar, M., Mahadevan, S., Breves, E., Lepore, K., and Clegg, S. (2015). A study of machine learning regression methods for major elemental analysis of rocks using laser-induced breakdown spectroscopy. Spectrochimica Acta - Part B Atomic Spectroscopy, 107:1–10. cited By 30.
Chang, C.-C. and Lin, C.-J. (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27. Software available at http://www.csie.ntu.edu.tw/˜cjlin/libsvm.
De Rivas, B., Vivancos, J.-L., Ordieres-Meré, J., and Capuz-Rizo, S. (2017). Determnation of the total acid number (tan) of used mineral oils in aviation engines by ftir using regression models. Chemometrics and Intelligent Laboratory Systems, 160:32– 39. cited By 6.
Drucker, H., Burges, C. J. C., Kaufman, L., Smola, A. J., and Vapnik, V. (1997). Support vector regression machines. In Mozer, M. C., Jordan, M. I., and Petsche, T., editors, Advances in Neural Information Processing Systems 9, pages 155–161. MIT Press.
Dua, D. and Graff, C. (2017). UCI machine learning repository.
Friedman, J., Hastie, T., and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1):1–22.
Gelius-Dietrich, G. (2017). cplexAPI: R Interface to C API of IBM ILOG CPLEX. R package version 1.3.3.
Jaggi, M. (2013). An equivalence between the lasso and support vector machines. Regularization, optimization, kernels, and support vector machines, chap 1.
Mercer, J. (1909). Functions of positive and negative type, and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 209:415– 446.
Meyer, D., Dimitriadou, E., Hornik, K., Weingessel, A., and Leisch, F. (2017). e1071: Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien. R package version 1.6-8.
Smola, A., Scholkopf, B., and Ratsch, G. (1999). Linear programs for automatic accuracy control in regression. In 1999 Ninth International Conference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470), volume 2, pages 575–580 vol.2.
Smola, A. J. and Schölkopf, B. (2004). A tutorial on support vector regression. Statistics and Computing, 14(3):199–222.
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58(1):267–288.
Tikhonov, A. N. (1963). On the solution of ill-posed problems and the method of regularization. Doklady Akademii Nauk SSSR, 151(3):501–504.
Zhou, Q., Chen, W., Song, S., Gardner, J. R., Weinberger, K. Q., and Chen, Y. (2015). A reduction of the elastic net to support vector machines with an application to gpu computing. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, pages 3210–3216. AAAI Press.
Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B, 67:301–320.
Published
2019-10-15
How to Cite
TEIXEIRA, Lucas; F. NETO, Raul.
Regularized Regression solution with Support Vectors through Linear Programming. In: NATIONAL MEETING ON ARTIFICIAL AND COMPUTATIONAL INTELLIGENCE (ENIAC), 16. , 2019, Salvador.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
p. 1032-1043.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2019.9355.
