Joint Learning of Sparse Gaussian Processes and Gaussian Process Latent Variable Models for Semi-supervised Tasks

• Ana Alice Ximenes Mota Peres UFC
• César Lincoln Cavalcante Mattos UFC

DOI: https://doi.org/10.5753/eniac.2025.14206

Abstract

The speed and variety of collected data have increased in a surprising way, corroborating with the creation of large and diverse datasets. When using such data for training supervised machine learning models, it is necessary to annotate the available samples. However, labeling instances can be challenging, expensive, and time-consuming. In this context, semi-supervised learning models have been extensively researched over the past decades. Among the supervised learning methods, models based on Gaussian Processes (GPs) offer the advantage of quantifying uncertainties and providing significant modeling flexibility. Nevertheless, like several learning strategies, they cannot be directly applied to semi-supervised scenarios. To overcome this issue, the current work proposes a GP-based approach to perform semi-supervised learning. The proposal consists in simultaneously training an unsupervised GP latent variable model (GPLVM) and a supervised sparse GP model. The approach leverages both labeled and unlabeled data to create a more effective final classifier. Additionally, a neural network is included to reproduce the latent variables learned by the GPLVM, which enables scaling and eases its use with unseen data. The introduced solution is evaluated on public datasets and compared with standard semi-supervised approaches from the literature, in both inductive and transductive settings. The experiments indicate that the proposed technique, despite some mixing results, is competitive, especially in transductive learning problems.

References

Bishop, C. M. and James, G. D. (1993). Analysis of multiphase flows using dualenergy gamma densitometry and neural networks. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 327(2-3):580–593.

Blum, A. and Mitchell, T. (1998). Combining labeled and unlabeled data with co-training. In Proceedings of the 11th Annual Conference on Learning Theory, pages 92–100.

Chapelle, O., Schölkopf, B., and Zien, A. (2006a). Introduction to semi-supervised learning. In Semi-Supervised Learning, pages 1–12. The MIT Press.

Chapelle, O., Schölkopf, B., and Zien, A., editors (2006b). Semi-Supervised Learning. The MIT Press, Massachusetts, USA.

Dai, Z., Damianou, A. C., González, J., and Lawrence, N. D. (2016). Variational autoencoded deep gaussian processes. In Proceedings of the 4th ICLR, San Juan.

Damianou, A. C. and Lawrence, N. D. (2015). Semi-described and semi-supervised learning with gaussian processes. In Proceedings of the 31st UAI, Amsterdam, The Netherlands, pages 228–237. AUAI Press.

Dua, D. and Graff, C. (2017). UCI machine learning repository.

Fujiwara, Y. and Irie, G. (2014). Efficient label propagation. In International conference on machine learning, pages 784–792. PMLR.

Gärtner, T., Le, Q., Burton, S., Smola, A. J., and Vishwanathan, V. (2005). Large-scale multiclass transduction. Advances in Neural Information Processing Systems, 18.

Hensman, J., Fusi, N., and Lawrence, N. D. (2013). Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI 2013, Bellevue, WA, USA, August 11-15, 2013. AUAI Press.

Kingma, D. P., Mohamed, S., Jimenez Rezende, D., and Welling, M. (2014). Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27.

Lalchand, V., Ravuri, A., and Lawrence, N. D. (2022). Generalised gplvm with stochastic variational inference. In International Conference on Artificial Intelligence and Statistics, pages 7841–7864. PMLR.

Lawrence, N. (2003). Gaussian process latent variable models for visualisation of high dimensional data. Advances in neural information processing systems, 16.

Lawrence, N. and Jordan, M. (2004). Semi-supervised learning via gaussian processes. Advances in neural information processing systems, 17.

Lawrence, N. D. and Quinonero-Candela, J. (2006). Local distance preservation in the gp-lvm through back constraints. In Proceedings of the 23rd international conference on Machine learning, pages 513–520.

Le, Q. V., Smola, A. J., Gärtner, T., and Altun, Y. (2006). Transductive gaussian process regression with automatic model selection. In Proceedings of the 17th ECML, Berlin, Germany, pages 306–317. Springer.

Li, H., Li, Y., and Lu, H. (2008). Semi-supervised learning with gaussian processes. In 2008 Chinese Conference on Pattern Recognition, pages 1–5. IEEE.

Mattos, C. L. C. and Barreto, G. A. (2019). A stochastic variational framework for Recurrent Gaussian Processes models. Neural Networks, 112:54–72.

Minka, T. P. (2001). Expectation propagation for approximate bayesian inference. In Proceedings of the 17th UAI, Seattle, USA, pages 362–369. Morgan Kaufmann.

Neal, R. (1999). Markov chain sampling using hamiltonian dynamics. In Talk at the Joint Statistical Meetings, Baltimore, August.

Prakash, V. J. and Nithya, D. L. (2014). A survey on semi-supervised learning techniques. arXiv preprint arXiv:1402.4645.

Quirion, S., Duchesne, C., Laurendeau, D., and Marchand, M. (2008). Comparing gplvm approaches for dimensionality reduction in character animation.

Rogers, S. and Girolami, M. (2007). Multi-class semi-supervised learning with the epsilon-truncated multinomial probit gaussian process. In Gaussian Processes in Practice, pages 17–32. PMLR.

Scudder, H. (1965). Adaptive communication receivers. IEEE Transactions on Information Theory, 11(2):167–174.

Sindhwani, V., Chu, W., and Keerthi, S. S. (2007). Semi-supervised gaussian process classifiers. In IJCAI, pages 1059–1064.

Snelson, E. and Ghahramani, Z. (2005). Sparse gaussian processes using pseudo-inputs. Advances in neural information processing systems, 18.

Srijith, P., Shevade, S., and Sundararajan, S. (2013). Semi-supervised gaussian process ordinal regression. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13, pages 144–159. Springer.

Street, W. N., Wolberg, W. H., and Mangasarian, O. L. (1993). Nuclear feature extraction for breast tumor diagnosis. In Biomedical image processing and biomedical visualization, volume 1905, pages 861–870. SPIE.

Titsias, M. (2009). Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics, pages 567–574. PMLR.

Titsias, M. and Lawrence, N. D. (2010). Bayesian gaussian process latent variable model. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 844–851. JMLR Workshop and Conference Proceedings.

Van Engelen, J. E. and Hoos, H. H. (2020). A survey on semi-supervised learning. Machine learning, 109(2):373–440.

Wainwright, M. J., Jordan, M. I., et al. (2008). Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1–2):1–305.

Williams, C. K. and Barber, D. (1998). Bayesian classification with gaussian processes. IEEE Transactions on pattern analysis and machine intelligence, 20(12):1342–1351.

Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA.

Zhou, D., Bousquet, O., Lal, T., Weston, J., and Schölkopf, B. (2003). Learning with local and global consistency. Advances in neural information processing systems, 16.

Zhu, X. and Ghahramani, Z. (2002). Learning from labeled and unlabeled data with label propagation. ProQuest number: information to all users.

Zouhal, L. M. and Denoeux, T. (1998). An evidence-theoretic k-nn rule with parameter optimization. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 28(2):263–271.