Previsão do tempo de resposta de aplicações de big data em ambientes de nuvem
Resumo
Aplicações de big data têm tipicamente propriedades bem específicas, tais como heterogeneidade e irregularidade nos padrões de acesso aos dados, que tornam a alocação de recursos de hardware e software muito desafiadora. Por outro lado, a flexibilidade e a elasticidade provida por plataformas de computação na nuvem facilitam esta alocaçãoá medida em que recursos são alocados sob demanda. Entretanto, estas características também tornam a previsão de desempenho (p.ex: tempo de resposta das aplicações) mais complexa. Este trabalho explora um modelo analítico para a previsão de tempo de resposta de aplicações executando na plataforma Spark, muito popular para processamento de dados em larga escala, que é parametrizado a partir de logs de execuções prévias. O modelo é avaliado em diversos cenários e aplicações, obtendo um erro relativo no tempo de resposta previsto inferior a 8%, em média.
Referências
Ardagna, D. et al. (2016). Modeling performance of hadoop applications: A journey from queueing networks to stochastic well formed nets. p. 599–613.
Bertoli, M., Casale, G., and Serazzi, G. (2009). JMT: performance engineering tools for system modeling. SIGMETRICS Performance Evaluation Review, 36(4). p. 10–15.
Chen, C. P. and Zhang, C.-Y. (2014). Data-intensive applications, challenges, techniques and technologies: A survey on big data. Information Sciences, 275. p. 314–347.
Chiola, G. (1985). A software package for the analysis of generalized stochastic petri net models. In International Workshop on Timed Petri Nets, Italy, 1985. p. 136–143. Spark Databricks (2016).
Apache Survey 2016. Disponível em: https://databricks.com/2016-spark-survey. Acessado em: 30/11/2017.
Herodotou, H. et al. (2011). Starsh: A self-tuning system for big data analytics. In Proceedings of the 5th CIDR, pages 261–272.
Laskowski, J. (2016). Mastering Apache Spark. Disponível em: https://gitbook.com/book/jaceklaskowski/mastering-apache-spark. Acessado em: 28/06/2017.
Mak, V. and Lundstrom, S. (1990). Predicting performance of parallel computations. IEEE Transactions on Parallel & Distributed Systems, 1. p. 257-270.
Menasce, D. A., Almeida, V. A., Dowdy, L. W., and Dowdy, L. (2004). Performance by design: computer capacity planning by example. Prentice Hall Professional.
Microsoft (2016). What is PaaS? Disponível em: https://azure.microsoft.com/enus/overview/what-is-paas/. Acessado em: 15/06/2017.
Nambiar, R. O. and Poess, M. (2006). The making of tpc-ds. In Proceedings of the 32Nd International Conference on Very Large Data Bases. p. 1049–1058.
Nelson, R. D. and Tantawi, A. N. (1988). Approximate analysis of fork/join synchronization in parallel queues. IEEE Trans. Computers, 37(6). p. 739-743.
Poess, M., Nambiar, R. O., and Walrath, D. (2007). Why you should run tpc-ds: A workload analysis. In Proceedings of the 33rd VLDB. p. 1138–1149.
Popescu, A. D. (2015). Runtime Prediction for Scale-Out Data Analytics. PhD thesis, íEcole Polytechnique Fédérale de Lausanne.
Reisig, W., Rozenberg, G., and Thiagarajan, P. S. (2013). In Memoriam: Carl Adam Petri. Springer Berlin Heidelberg, Berlin, Heidelberg. p. 1–5.
Song, G., Meng, Z., Huet, F., Magoules, F., Yu, L., and et al. (2013). A hadoop mapreduce performance prediction method. In Proceedings of the HPCC 2013. p. 820-825.
Tripathi, S. K. and Liang, D.-R. (2000). On performance prediction of parallel computations with precedent constraints. IEEE TPDS, 11. p. 491-508.
Truta, H., Vivas, J. L., Brito, A., and Nobrega, T. (2017). A predictive approach for enhancing resource utilization in paas clouds. In Proceedings of the SAC 2017.
Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., and Stoica, I. (2010). Spark: Cluster computing with working sets. In Proceedings of the 2Nd USENIX HotCloud.
Zaharia, M. et al. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX NSDI.
Zaki, M. J., Meira Jr, W., and Meira, W. (2014). Data mining and analysis: fundamental concepts and algorithms. Cambridge University Press.
Bertoli, M., Casale, G., and Serazzi, G. (2009). JMT: performance engineering tools for system modeling. SIGMETRICS Performance Evaluation Review, 36(4). p. 10–15.
Chen, C. P. and Zhang, C.-Y. (2014). Data-intensive applications, challenges, techniques and technologies: A survey on big data. Information Sciences, 275. p. 314–347.
Chiola, G. (1985). A software package for the analysis of generalized stochastic petri net models. In International Workshop on Timed Petri Nets, Italy, 1985. p. 136–143. Spark Databricks (2016).
Apache Survey 2016. Disponível em: https://databricks.com/2016-spark-survey. Acessado em: 30/11/2017.
Herodotou, H. et al. (2011). Starsh: A self-tuning system for big data analytics. In Proceedings of the 5th CIDR, pages 261–272.
Laskowski, J. (2016). Mastering Apache Spark. Disponível em: https://gitbook.com/book/jaceklaskowski/mastering-apache-spark. Acessado em: 28/06/2017.
Mak, V. and Lundstrom, S. (1990). Predicting performance of parallel computations. IEEE Transactions on Parallel & Distributed Systems, 1. p. 257-270.
Menasce, D. A., Almeida, V. A., Dowdy, L. W., and Dowdy, L. (2004). Performance by design: computer capacity planning by example. Prentice Hall Professional.
Microsoft (2016). What is PaaS? Disponível em: https://azure.microsoft.com/enus/overview/what-is-paas/. Acessado em: 15/06/2017.
Nambiar, R. O. and Poess, M. (2006). The making of tpc-ds. In Proceedings of the 32Nd International Conference on Very Large Data Bases. p. 1049–1058.
Nelson, R. D. and Tantawi, A. N. (1988). Approximate analysis of fork/join synchronization in parallel queues. IEEE Trans. Computers, 37(6). p. 739-743.
Poess, M., Nambiar, R. O., and Walrath, D. (2007). Why you should run tpc-ds: A workload analysis. In Proceedings of the 33rd VLDB. p. 1138–1149.
Popescu, A. D. (2015). Runtime Prediction for Scale-Out Data Analytics. PhD thesis, íEcole Polytechnique Fédérale de Lausanne.
Reisig, W., Rozenberg, G., and Thiagarajan, P. S. (2013). In Memoriam: Carl Adam Petri. Springer Berlin Heidelberg, Berlin, Heidelberg. p. 1–5.
Song, G., Meng, Z., Huet, F., Magoules, F., Yu, L., and et al. (2013). A hadoop mapreduce performance prediction method. In Proceedings of the HPCC 2013. p. 820-825.
Tripathi, S. K. and Liang, D.-R. (2000). On performance prediction of parallel computations with precedent constraints. IEEE TPDS, 11. p. 491-508.
Truta, H., Vivas, J. L., Brito, A., and Nobrega, T. (2017). A predictive approach for enhancing resource utilization in paas clouds. In Proceedings of the SAC 2017.
Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., and Stoica, I. (2010). Spark: Cluster computing with working sets. In Proceedings of the 2Nd USENIX HotCloud.
Zaharia, M. et al. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX NSDI.
Zaki, M. J., Meira Jr, W., and Meira, W. (2014). Data mining and analysis: fundamental concepts and algorithms. Cambridge University Press.
Publicado
10/05/2018
Como Citar
PINTO, Túlio B. M.; SILVA, Ana Paula Couto da; ALMEIDA, Jussara M..
Previsão do tempo de resposta de aplicações de big data em ambientes de nuvem. In: SIMPÓSIO BRASILEIRO DE REDES DE COMPUTADORES E SISTEMAS DISTRIBUÍDOS (SBRC), 36. , 2018, Campos do Jordão.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2018
.
p. 533-546.
ISSN 2177-9384.
DOI: https://doi.org/10.5753/sbrc.2018.2440.
