Semi-Automatic Data Annotation guided by Feature Space Projection
Resumo
Data annotation using visual inspection (supervision) of each training sample can be laborious. Interactive solutions alleviate this by helping experts propagate labels from a few supervised samples to unlabeled ones based solely on the visual analysis of their feature space projection (with no further sample supervision). We present a semi-automatic data annotation approach based on suitable feature space projection and semi-supervised label estimation. We validate our method on the popular MNIST dataset and on images of human intestinal parasites with and without fecal impurities, a large and diverse dataset that makes classification very hard. We evaluate two approaches for semi-supervised learning from the latent and projection spaces, to choose the one that best reduces user annotation effort and also increases classification accuracy on unseen data. Our results demonstrate the added-value of visual analytics tools that combine complementary abilities of humans and machines for more effective machine learning.Referências
B. Benato, "Anotaçãao de dados guiada por projeção de características," Instituto de Master’s thesis, Universidade Estadual de Campinas, Computação, Campinas, SP., 2019.
A. Krizhevsky, I. Sutskever, and H. E. Geoffrey, "Imagenet classification with deep convolutional neural networks," in Advances in Neural Infor- mation Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2012, pp. 1097–1105.
J. Kim, J. K. Lee, and K. M. Lee, "Accurate image super-resolution using very deep convolutional networks," in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-end training of deep visuomotor policies," J. Mach. Learn. Res., vol. 17, no. 1, pp. 1334–1373, Jan. 2016.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, Nov 1998.
G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, Nov 2012.
J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?" in Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, ser. NIPS’14. Cambridge, MA, USA: MIT Press, 2014, pp. 3320–3328.
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, Jan. 2014.
R. Mash, B. Borghetti, and J. Pecarina, "Improved aircraft recognition for aerial refueling through data augmentation in convolutional neural networks," in Proc. ISVC. Springer, 2016, pp. 113–122.
S. J. Nowlan and G. E. Hinton, "Simplifying neural networks by soft weight-sharing," Neural Computation, vol. 4, no. 4, pp. 473–493, 1992.
D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling, "Semi- supervised learning with deep generative models," in Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, Curran C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds. Associates, Inc., 2014, pp. 3581–3589.
G. Forestier and C. Wemmert, "Semi-supervised learning using multiple clusterings with limited labeled data," Information Sciences, vol. 361– 362, pp. 48–65, 2016.
N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar, "Semi-supervised knowledge transfer for deep learning from private training data," in Proceedings of the International Conference on Learning Representations, 2017.
P. Rauber, A. Falcão, and A. Telea, "Projections as visual aids for classification system design," Information Visualization, 2017.
P. E. Rauber, A. X. Falcão, and A. C. Telea, "Visualizing time-dependent data using dynamic t-SNE," in Proceedings of the Eurographics / IEEE VGTC Conference on Visualization: Short Papers, ser. EuroVis ’16, 2016, pp. 73–77.
A. Z. Peixinho, B. C. Benato, L. G. Nonato, and A. X. Falcão, "Delaunay triangulation data augmentation guided by visual analytics for deep learning," in 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2018, pp. 384–391.
J. Bernard, M. Hutter, M. Zeppelzauer, D. Fellner, and M. Sedlmair, "Comparing visual-interactive labeling with active learning: An experimental study," IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 1, pp. 298–308, Jan 2018.
B. C. Benato, A. C. Telea, and A. X. Falcão, "Semi-supervised learning with interactive label propagation guided by feature space projections," in 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2018, pp. 392–399.
B. Settles, "Active learning literature survey," University of Wisconsin– Madison, Computer Sciences Technical Report 1648, 2009.
S. Patra and L. Bruzzone, "A batch-mode active learning technique based on multiple uncertainty for SVM classifier," IEEE Geoscience and Remote Sensing Letters, vol. 9, no. 3, pp. 497–501, May 2012.
P. A. V. Miranda and A. X. Falcão, "Links between image segmentation based on optimum-path forest and minimum cut in graph," Journal of Mathematical Imaging and Vision, vol. 35, no. 2, pp. 128–142, Oct 2009.
T. V. Spina, P. A. V. Miranda, and A. X. Falcão, "Intelligent understanding of user interaction in image segmentation," International Journal of Pattern Recognition and Artificial Intelligence, vol. 26, no. 02, p. 1265001, 2012.
A. T. Silva, J. A. Santos, A. X. Falcao, R. S. Torres, and L. P. Magalhaes, "Incorporating multiple distance spaces in optimum-path forest classification to improve feedback-based learning," Computer Vision and Image Understanding, vol. 116, no. 4, pp. 510 – 523, 2012.
L. V. D. Maaten, "Accelerating t-SNE using tree-based algorithms," Journal of Machine Learning Research, vol. 15, no. 1, pp. 3221–3245, 2014.
B. C. Benato, J. F. Gomes, A. C. Telea, and A. X. Falcão, "Semi- automatic data annotation guided by feature space projection," Pattern Recognition, vol. 109, p. 107612, 2021.
J. Masci, U. Meier, D. Cires¸an, and J. Schmidhuber, "Stacked convolutional auto-encoders for hierarchical feature extraction," in Proc. Intl. Conf. on Artificial Neural Networks (ICANN). Springer, 2011, pp. 52– 59.
P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. A. Manzagol, "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion," J. Mach. Learn. Res., vol. 11, pp. 3371–3408, Dec. 2010.
G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504–507, Jul. 2006.
L. van der Maaten and G. Hinton, "Visualizing data using t-SNE," Journal of Machine Learning Research, vol. 9, pp. 2579–2605, 2008.
V. Sindhwani, P. Niyogi, and M. Belkin, "Beyond the point cloud: From transductive to semi-supervised learning," in Proceedings of the 22Nd International Conference on Machine Learning, ser. ICML ’05. New York, NY, USA: ACM, 2005, pp. 824–831.
M. Belkin, P. Niyogi, and V. Sindhwani, "Manifold regularization: A geometric framework for learning from labeled and unlabeled examples," J. Mach. Learn. Res., vol. 7, pp. 2399–2434, Dec. 2006.
W. P. Amorim, A. X. Falcão, J. P. Papa, and M. H. Carvalho, "Improving semi-supervised learning through optimum connectivity," Pattern Recogn., vol. 60, no. C, pp. 72–85, Dec. 2016.
L. Nonato and M. Aupetit, "Multidimensional projection for visual analytics: Linking techniques with distortions, tasks, and layout enrichment," IEEE TVCG, 2018.
J. P. Papa, A. X. Falcao, V. H. C. Albuquerque, and J. M. R. S. Tavares, "Efficient supervised optimum-path forest classification for large datasets," Pattern Recognition, vol. 45, no. 1, pp. 512 – 520, 2012.
M. A. Hearst, S. T. Dumais, E. Osuna, J. Platt, and B. Scholkopf, "Support vector machines," IEEE Intelligent Systems and their Applications, vol. 13, no. 4, pp. 18–28, July 1998.
Y. LeCun and C. Cortes, "MNIST handwritten digit database," 2010. [Online]. Available: http://yann.lecun.com/exdb/mnist/
C. T. N. Suzuki, J. F. Gomes, A. X. Falcão, S. H. Shimizu, and J. P. Papa, "Automated diagnosis of human intestinal parasites using optical microscopy images," in 2013 IEEE 10th International Symposium on Biomedical Imaging, April 2013, pp. 460–463.
S. B. Martins, B. C. Benato, B. F. Silva, C. L. Yasuda, and A. X. Falcão, "Modeling normal brain asymmetry in MR images applied to anomaly detection without segmentation and data annotation," in Imaging 2019: Computer-Aided Diagnosis, K. Mori and Medical International Society for Optics H. K. Hahn, Eds., vol. 10950, and Photonics. [Online]. Available: https://doi.org/10.1117/12.2512873 SPIE, 2019, pp. 71 – 80.
A. Krizhevsky, I. Sutskever, and H. E. Geoffrey, "Imagenet classification with deep convolutional neural networks," in Advances in Neural Infor- mation Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2012, pp. 1097–1105.
J. Kim, J. K. Lee, and K. M. Lee, "Accurate image super-resolution using very deep convolutional networks," in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
S. Levine, C. Finn, T. Darrell, and P. Abbeel, "End-to-end training of deep visuomotor policies," J. Mach. Learn. Res., vol. 17, no. 1, pp. 1334–1373, Jan. 2016.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, Nov 1998.
G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, Nov 2012.
J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?" in Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, ser. NIPS’14. Cambridge, MA, USA: MIT Press, 2014, pp. 3320–3328.
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, Jan. 2014.
R. Mash, B. Borghetti, and J. Pecarina, "Improved aircraft recognition for aerial refueling through data augmentation in convolutional neural networks," in Proc. ISVC. Springer, 2016, pp. 113–122.
S. J. Nowlan and G. E. Hinton, "Simplifying neural networks by soft weight-sharing," Neural Computation, vol. 4, no. 4, pp. 473–493, 1992.
D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling, "Semi- supervised learning with deep generative models," in Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, Curran C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds. Associates, Inc., 2014, pp. 3581–3589.
G. Forestier and C. Wemmert, "Semi-supervised learning using multiple clusterings with limited labeled data," Information Sciences, vol. 361– 362, pp. 48–65, 2016.
N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar, "Semi-supervised knowledge transfer for deep learning from private training data," in Proceedings of the International Conference on Learning Representations, 2017.
P. Rauber, A. Falcão, and A. Telea, "Projections as visual aids for classification system design," Information Visualization, 2017.
P. E. Rauber, A. X. Falcão, and A. C. Telea, "Visualizing time-dependent data using dynamic t-SNE," in Proceedings of the Eurographics / IEEE VGTC Conference on Visualization: Short Papers, ser. EuroVis ’16, 2016, pp. 73–77.
A. Z. Peixinho, B. C. Benato, L. G. Nonato, and A. X. Falcão, "Delaunay triangulation data augmentation guided by visual analytics for deep learning," in 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2018, pp. 384–391.
J. Bernard, M. Hutter, M. Zeppelzauer, D. Fellner, and M. Sedlmair, "Comparing visual-interactive labeling with active learning: An experimental study," IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 1, pp. 298–308, Jan 2018.
B. C. Benato, A. C. Telea, and A. X. Falcão, "Semi-supervised learning with interactive label propagation guided by feature space projections," in 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Oct 2018, pp. 392–399.
B. Settles, "Active learning literature survey," University of Wisconsin– Madison, Computer Sciences Technical Report 1648, 2009.
S. Patra and L. Bruzzone, "A batch-mode active learning technique based on multiple uncertainty for SVM classifier," IEEE Geoscience and Remote Sensing Letters, vol. 9, no. 3, pp. 497–501, May 2012.
P. A. V. Miranda and A. X. Falcão, "Links between image segmentation based on optimum-path forest and minimum cut in graph," Journal of Mathematical Imaging and Vision, vol. 35, no. 2, pp. 128–142, Oct 2009.
T. V. Spina, P. A. V. Miranda, and A. X. Falcão, "Intelligent understanding of user interaction in image segmentation," International Journal of Pattern Recognition and Artificial Intelligence, vol. 26, no. 02, p. 1265001, 2012.
A. T. Silva, J. A. Santos, A. X. Falcao, R. S. Torres, and L. P. Magalhaes, "Incorporating multiple distance spaces in optimum-path forest classification to improve feedback-based learning," Computer Vision and Image Understanding, vol. 116, no. 4, pp. 510 – 523, 2012.
L. V. D. Maaten, "Accelerating t-SNE using tree-based algorithms," Journal of Machine Learning Research, vol. 15, no. 1, pp. 3221–3245, 2014.
B. C. Benato, J. F. Gomes, A. C. Telea, and A. X. Falcão, "Semi- automatic data annotation guided by feature space projection," Pattern Recognition, vol. 109, p. 107612, 2021.
J. Masci, U. Meier, D. Cires¸an, and J. Schmidhuber, "Stacked convolutional auto-encoders for hierarchical feature extraction," in Proc. Intl. Conf. on Artificial Neural Networks (ICANN). Springer, 2011, pp. 52– 59.
P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. A. Manzagol, "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion," J. Mach. Learn. Res., vol. 11, pp. 3371–3408, Dec. 2010.
G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504–507, Jul. 2006.
L. van der Maaten and G. Hinton, "Visualizing data using t-SNE," Journal of Machine Learning Research, vol. 9, pp. 2579–2605, 2008.
V. Sindhwani, P. Niyogi, and M. Belkin, "Beyond the point cloud: From transductive to semi-supervised learning," in Proceedings of the 22Nd International Conference on Machine Learning, ser. ICML ’05. New York, NY, USA: ACM, 2005, pp. 824–831.
M. Belkin, P. Niyogi, and V. Sindhwani, "Manifold regularization: A geometric framework for learning from labeled and unlabeled examples," J. Mach. Learn. Res., vol. 7, pp. 2399–2434, Dec. 2006.
W. P. Amorim, A. X. Falcão, J. P. Papa, and M. H. Carvalho, "Improving semi-supervised learning through optimum connectivity," Pattern Recogn., vol. 60, no. C, pp. 72–85, Dec. 2016.
L. Nonato and M. Aupetit, "Multidimensional projection for visual analytics: Linking techniques with distortions, tasks, and layout enrichment," IEEE TVCG, 2018.
J. P. Papa, A. X. Falcao, V. H. C. Albuquerque, and J. M. R. S. Tavares, "Efficient supervised optimum-path forest classification for large datasets," Pattern Recognition, vol. 45, no. 1, pp. 512 – 520, 2012.
M. A. Hearst, S. T. Dumais, E. Osuna, J. Platt, and B. Scholkopf, "Support vector machines," IEEE Intelligent Systems and their Applications, vol. 13, no. 4, pp. 18–28, July 1998.
Y. LeCun and C. Cortes, "MNIST handwritten digit database," 2010. [Online]. Available: http://yann.lecun.com/exdb/mnist/
C. T. N. Suzuki, J. F. Gomes, A. X. Falcão, S. H. Shimizu, and J. P. Papa, "Automated diagnosis of human intestinal parasites using optical microscopy images," in 2013 IEEE 10th International Symposium on Biomedical Imaging, April 2013, pp. 460–463.
S. B. Martins, B. C. Benato, B. F. Silva, C. L. Yasuda, and A. X. Falcão, "Modeling normal brain asymmetry in MR images applied to anomaly detection without segmentation and data annotation," in Imaging 2019: Computer-Aided Diagnosis, K. Mori and Medical International Society for Optics H. K. Hahn, Eds., vol. 10950, and Photonics. [Online]. Available: https://doi.org/10.1117/12.2512873 SPIE, 2019, pp. 71 – 80.
Publicado
07/11/2020
Como Citar
BENATO, Bárbara C.; TELEA, Alexandru C.; FALCÃO, Alexandre X..
Semi-Automatic Data Annotation guided by Feature Space Projection. In: WORKSHOP DE TESES E DISSERTAÇÕES - CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 33. , 2020, Evento Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2020
.
p. 1-7.
DOI: https://doi.org/10.5753/sibgrapi.est.2020.12976.
