Análise Experimental da Escalabilidade e Elasticidade de Bibliotecas de Análise de Dados em Workflows com Alto Custo Computacional
Resumo
Este trabalho analisa experimentalmente a escalabilidade e a elasticidade de bibliotecas Python para análise de dados a Pandas, a Polars, a Dask, a DuckDB e a PySpark em diferentes volumes e configurações de recursos. Os resultados mostram que nenhuma das bibliotecas apresentou escalabilidade plena nos cenários testados: todas exibiram speedup sublinear com saturação, com eficiência decrescente conforme novos núcleos eram adicionados; a DuckDB obteve os menores tempos absolutos de execução e a Polars as maiores médias de speedup e eficiência, enquanto a PySpark, a Pandas e a Dask apresentaram ganhos mais limitados. Na elasticidade, as bibliotecas mostraram-se elásticas horizontalmente, com a PySpark aproveitando melhor os recursos alocados dinamicamente (mais próximo do teto configurado e com escalonamento mais estável) e a Dask alcançando os menores tempos de execução, embora, em ambos os casos, um teto maior de workers tenha reduzido o tempo total.Referências
Balusamy, B., Abirami R, N., Kadry, S., and Gandomi, A. H. (2021). Big Data: Concepts, Technology, and Architecture. Wiley.
Dask Development Team (2026). Dask documentation. Acessado em: 5 de agosto de 2026.
Drabas, T. and Lee, D. (2017). Learning PySpark. Packt Publishing, Birmingham, UK.
Gu, R., Shi, J., Chen, X., Wang, Z., Che, Y., Zhang, K., and Huang, Y. (2022). Octopus-df: Unified dataframe-based cross-platform data analytic system. Parallel Computing, 110:102879. Acessado em: 5 de agosto de 2026.
Herbst, N. R., Kounev, S., and Reussner, R. (2013). Elasticity in cloud computing: What it is, and what it is not. Proceedings of the 10th International Conference on Autonomic Computing (ICAC ’13), pages 23–27.
Kubernetes (2026). Kubernetes documentation. Acesso em: 1 Ago 2026.
Lehrig, S., Eikerling, H., and Becker, S. (2015). Scalability, elasticity, and efficiency in cloud computing: A systematic literature review of definitions and metrics. Proceedings of the 11th International ACM SIGSOFT Conference on Quality of Software Architectures (QoSA ’15), pages 83–92.
McKinney, W. (2023). Python para Análise de Dados: Tratamento de Dados com pandas, NumPy & Jupyter. Novatec Editora, São Paulo, 3 edition.
Medel, V. et al. (2024). Characterising resource management performance in kubernetes. arXiv preprint arXiv:2401.17125.
Mozzillo, A., Zecchini, L., Gagliardelli, L., Aslam, A., Bergamaschi, S., and Simonini, G. (2025). Evaluation of dataframe libraries for data preparation on a single machine.
Nahrstedt, F., Karmouche, M., Bargieł, K., Banijamali, P., Nalini Pradeep Kumar, A., and Malavolta, I. (2024). An empirical study on the energy usage and performance of pandas and polars data analysis python libraries. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, EASE ’24, page 58–68, New York, NY, USA. Association for Computing Machinery.
Petersohn, D. (2018). Scaling interactive data science transparently with modin. Master’s thesis, EECS Department, University of California, Berkeley.
Piñeiro, Á. M. (2022). Big data para la comprensión de métricas de e-sports usando tecnologías web. Master’s thesis, E.T.S. de Ingenieros Informáticos, Universidad Politécnica de Madrid, Madrid. Acessado em: 5 de agosto de 2026.
Putrama, I. M. and Martinek, P. (2024). Heterogeneous data integration: Challenges and opportunities. Data in Brief, 56:110853.
Rocklin, M. (2015). Dask: Parallel computation with blocked algorithms and task scheduling. Proceedings of the 14th Python in Science Conference, pages 130–136.
Transaction Processing Performance Council (2023). Tpc-h benchmark specification. TPC Official Documentation.
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M. J., Shenker, S., and Stoica, I. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. Proc. 9th USENIX NSDI, pages 15–28.
Dask Development Team (2026). Dask documentation. Acessado em: 5 de agosto de 2026.
Drabas, T. and Lee, D. (2017). Learning PySpark. Packt Publishing, Birmingham, UK.
Gu, R., Shi, J., Chen, X., Wang, Z., Che, Y., Zhang, K., and Huang, Y. (2022). Octopus-df: Unified dataframe-based cross-platform data analytic system. Parallel Computing, 110:102879. Acessado em: 5 de agosto de 2026.
Herbst, N. R., Kounev, S., and Reussner, R. (2013). Elasticity in cloud computing: What it is, and what it is not. Proceedings of the 10th International Conference on Autonomic Computing (ICAC ’13), pages 23–27.
Kubernetes (2026). Kubernetes documentation. Acesso em: 1 Ago 2026.
Lehrig, S., Eikerling, H., and Becker, S. (2015). Scalability, elasticity, and efficiency in cloud computing: A systematic literature review of definitions and metrics. Proceedings of the 11th International ACM SIGSOFT Conference on Quality of Software Architectures (QoSA ’15), pages 83–92.
McKinney, W. (2023). Python para Análise de Dados: Tratamento de Dados com pandas, NumPy & Jupyter. Novatec Editora, São Paulo, 3 edition.
Medel, V. et al. (2024). Characterising resource management performance in kubernetes. arXiv preprint arXiv:2401.17125.
Mozzillo, A., Zecchini, L., Gagliardelli, L., Aslam, A., Bergamaschi, S., and Simonini, G. (2025). Evaluation of dataframe libraries for data preparation on a single machine.
Nahrstedt, F., Karmouche, M., Bargieł, K., Banijamali, P., Nalini Pradeep Kumar, A., and Malavolta, I. (2024). An empirical study on the energy usage and performance of pandas and polars data analysis python libraries. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, EASE ’24, page 58–68, New York, NY, USA. Association for Computing Machinery.
Petersohn, D. (2018). Scaling interactive data science transparently with modin. Master’s thesis, EECS Department, University of California, Berkeley.
Piñeiro, Á. M. (2022). Big data para la comprensión de métricas de e-sports usando tecnologías web. Master’s thesis, E.T.S. de Ingenieros Informáticos, Universidad Politécnica de Madrid, Madrid. Acessado em: 5 de agosto de 2026.
Putrama, I. M. and Martinek, P. (2024). Heterogeneous data integration: Challenges and opportunities. Data in Brief, 56:110853.
Rocklin, M. (2015). Dask: Parallel computation with blocked algorithms and task scheduling. Proceedings of the 14th Python in Science Conference, pages 130–136.
Transaction Processing Performance Council (2023). Tpc-h benchmark specification. TPC Official Documentation.
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M. J., Shenker, S., and Stoica, I. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. Proc. 9th USENIX NSDI, pages 15–28.
Publicado
03/11/2026
Como Citar
CIRILLO, Gabriel R. B.; GALANTE, Guilherme.
Análise Experimental da Escalabilidade e Elasticidade de Bibliotecas de Análise de Dados em Workflows com Alto Custo Computacional. In: SIMPÓSIO EM SISTEMAS COMPUTACIONAIS DE ALTO DESEMPENHO (SSCAD), 27. , 2026, Natal/RN.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 153-164.
DOI: https://doi.org/10.5753/sscad.2026.34886.
