SmarT: Uso de Aprendizado de Máquina para Filtragem e Recuperação Eficiente de Dados Espaciais e Temporais em Big Data
Resumo
Com o aumento do volume de dados de séries temporais na era de Big Data, a filtragem e recuperação eficiente de um grande volume de dados, utilizados como entrada no processamento de séries temporais, são uns dos maiores desafios da área. Diversos sistemas de Big Data foram criados para lidar com estes desafios, mas nenhum possui o melhor desempenho de filtragem e recuperação de dados em todos os cenários com filtros espaciais e temporais. Este trabalho apresenta o motor de busca SmarT que utiliza algoritmos de aprendizado de máquina para escolher, em tempo real, o melhor sistema Big Data para filtrar e recuperar os dados de séries temporais. O trabalho avalia o Apache Spark, Elasticsearch e SciDB e mostra uma redução de quase 22% do tempo de resposta utilizando o SmarT.
Palavras-chave:
Big Data, Inteligência Artificial, Séries Temporais
Referências
E. Calikus, S. Nowaczyk, A. Sant’Anna, and O. Dikmen. No free lunch but a cheaper supper: A general framework for streaming anomaly detection. Expert Systems with Applications, 155:113453, October 2020. ISSN 0957-4174.
R. Carmona. Statistical Analysis of Financial Data in R. Springer-Verlag New York, 2014.
H. Chen and N. Zhang. Graph-based change-point detection. The Annals of Statistics, 43(1):139–176, February 2015. ISSN 0090-5364.
D. De Paepe, S. V. Hautte, B. Steenwinckel, F. De Turck, F. Ongenae, O. Janssens, and S. V. Hoecke. A generalized matrix profile framework with support for contextual series analysis. Engineering Applications of Artificial Intelligence, 90, 2020. ISSN 0952-1976.
B. Eriksson, P. Barford, R. Bowden, N. Duffield, J. Sommers, and M. Roughan. Basisdetect: A model-based network event detection framework. In Proceedings of the 10th ACM SIGCOMM, page 451–464, New York, NY, USA, 2010. Association for Computing Machinery.
M. Gupta, J. Gao, C. Aggarwal, and J. Han. Outlier Detection for Temporal Data: A Survey. IEEE Transactions on Knowledge and Data Engineering, 2014. ISSN 1041-4347.
V. Guralnik and J. Srivastava. Event Detection from Time Series Data. In Proceedings of the Fifth ACM SIGKDD, KDD ’99, pages 33–42, New York, NY, USA, 1999. ACM. ISBN 978-1-58113-143-7.
G. Lu, Y. Zhou, C. Lu, and X. Li. A novel framework of change-point detection for machine monitoring. Mechanical Systems and Signal Processing, 83, 07 2016.
H. Raza, G. Prasad, and Y. Li. EWMA Model Based Shift-Detection Methods for Detecting Covariate Shifts in Non-Stationary Environments. Pattern Recogn., 48(3):659–669, March 2015. ISSN 0031-3203.
F. Rehbach, S. Moritz, S. Chandrasekaran, M. Rebolledo, M. Friese, and T. Bartz-Beielstein. GECCO 2018 Industrial Challenge: Monitoring of drinking-water quality. pages 1–7, 2018.
R. Salles, K. Belloze, F. Porto, P.H. Gonzalez, and E. Ogasawara. Nonstationary time series transformation methods: An experimental review. Knowledge-Based Systems, 164:274–291, 2019.
D. Silva, A. Simões, C. Cardoso, D. E. M. Oliveira, Y. Souto, L. E. G. Vignoli, R. Salles, H. S. C. Jr, A. Ziviani, E. Ogasawara, F. C. Delicato, P. F. Pires, H. L. C. P. Pinto, L. Maia, and F. Porto. A conceptual vision toward the management of machine learning models. In Proceedings of the ER Forum 2019, Salvador, Bahia, Brazil, volume 2469, pages 15–27, 2019.
J.-I. Takeuchi and K. Yamanishi. A unifying framework for detecting outliers and change points from time series. IEEE Transactions on Knowledge and Data Engineering, 18(4):482–492, 2006.
P. D. Talagala, R. J. Hyndman, K. Smith-Miles, S. Kandanaarachchi, and M. Muñoz. Anomaly Detection in Streaming Nonstationary Temporal Data. Journal of Computational and Graphical Statistics, 29(1):13–27, 2020.
Yahoo! Webscope. Labeled anomaly detection dataset. March 2015.
L. Xiong, C. Jiang, C. Xu, K. Yu, and S. Guo. A framework of change-point detection for multivariate hydrological series. Water Resources Research, 51, 09 2015.
R. Carmona. Statistical Analysis of Financial Data in R. Springer-Verlag New York, 2014.
H. Chen and N. Zhang. Graph-based change-point detection. The Annals of Statistics, 43(1):139–176, February 2015. ISSN 0090-5364.
D. De Paepe, S. V. Hautte, B. Steenwinckel, F. De Turck, F. Ongenae, O. Janssens, and S. V. Hoecke. A generalized matrix profile framework with support for contextual series analysis. Engineering Applications of Artificial Intelligence, 90, 2020. ISSN 0952-1976.
B. Eriksson, P. Barford, R. Bowden, N. Duffield, J. Sommers, and M. Roughan. Basisdetect: A model-based network event detection framework. In Proceedings of the 10th ACM SIGCOMM, page 451–464, New York, NY, USA, 2010. Association for Computing Machinery.
M. Gupta, J. Gao, C. Aggarwal, and J. Han. Outlier Detection for Temporal Data: A Survey. IEEE Transactions on Knowledge and Data Engineering, 2014. ISSN 1041-4347.
V. Guralnik and J. Srivastava. Event Detection from Time Series Data. In Proceedings of the Fifth ACM SIGKDD, KDD ’99, pages 33–42, New York, NY, USA, 1999. ACM. ISBN 978-1-58113-143-7.
G. Lu, Y. Zhou, C. Lu, and X. Li. A novel framework of change-point detection for machine monitoring. Mechanical Systems and Signal Processing, 83, 07 2016.
H. Raza, G. Prasad, and Y. Li. EWMA Model Based Shift-Detection Methods for Detecting Covariate Shifts in Non-Stationary Environments. Pattern Recogn., 48(3):659–669, March 2015. ISSN 0031-3203.
F. Rehbach, S. Moritz, S. Chandrasekaran, M. Rebolledo, M. Friese, and T. Bartz-Beielstein. GECCO 2018 Industrial Challenge: Monitoring of drinking-water quality. pages 1–7, 2018.
R. Salles, K. Belloze, F. Porto, P.H. Gonzalez, and E. Ogasawara. Nonstationary time series transformation methods: An experimental review. Knowledge-Based Systems, 164:274–291, 2019.
D. Silva, A. Simões, C. Cardoso, D. E. M. Oliveira, Y. Souto, L. E. G. Vignoli, R. Salles, H. S. C. Jr, A. Ziviani, E. Ogasawara, F. C. Delicato, P. F. Pires, H. L. C. P. Pinto, L. Maia, and F. Porto. A conceptual vision toward the management of machine learning models. In Proceedings of the ER Forum 2019, Salvador, Bahia, Brazil, volume 2469, pages 15–27, 2019.
J.-I. Takeuchi and K. Yamanishi. A unifying framework for detecting outliers and change points from time series. IEEE Transactions on Knowledge and Data Engineering, 18(4):482–492, 2006.
P. D. Talagala, R. J. Hyndman, K. Smith-Miles, S. Kandanaarachchi, and M. Muñoz. Anomaly Detection in Streaming Nonstationary Temporal Data. Journal of Computational and Graphical Statistics, 29(1):13–27, 2020.
Yahoo! Webscope. Labeled anomaly detection dataset. March 2015.
L. Xiong, C. Jiang, C. Xu, K. Yu, and S. Guo. A framework of change-point detection for multivariate hydrological series. Water Resources Research, 51, 09 2015.
Publicado
28/09/2020
Como Citar
OLIVEIRA, Sávio S. Teles de; RODRIGUES, Vagner J. do Sacramento; MARTINS, Wellington S..
SmarT: Uso de Aprendizado de Máquina para Filtragem e Recuperação Eficiente de Dados Espaciais e Temporais em Big Data. In: SIMPÓSIO BRASILEIRO DE BANCO DE DADOS (SBBD), 35. , 2020, Evento Online.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2020
.
p. 85-96.
ISSN 2763-8979.
DOI: https://doi.org/10.5753/sbbd.2020.13627.
