Scientific Workflow Deployment: Container Provenance in High-Performance Computing

• Liliane Kunstmann Universidade Federal do Rio de Janeiro
• Débora Pina Universidade Federal do Rio de Janeiro
• Daniel de Oliveira Universidade Federal Fluminense
• Marta Mattoso Universidade Federal do Rio de Janeiro

DOI: https://doi.org/10.5753/sbbd.2024.240194

Resumo

Deploying scientific workflows in high-performance computing (HPC) environments is increasingly challenging due to diverse computational settings. Containers help deploy and reproduce workflows, but both require more than just accessing container images. Container provenance provides essential information about image usage, origins, and recipes, crucial for deployment on various architectures or engines. Current support is limited to container actions and processes without workflow traceability. We propose extending workflow provenance to include container data using ProvDeploy, which supports workflow deployment with various container compositions in HPC, using W3C-PROV for container representation. We evaluated this with a real scientific machine learning workflow.

Palavras-chave: Container, Provenance, Workflows, Machine Learning

Referências

Abbas, M., Khan, S., Monum, A., Zaffar, F., et al. (2022). Paced: Provenance-based automated container escape detection. In 2022 IEEE IC2E, pages 261–272. IEEE.

Ahmad, R., Nakamura, Y., Manne, N. N., and Malik, T. (2020). PROV-CRT: Provenance support for container runtimes. In 12th International Workshop on Theory and Practice of Provenance (TaPP 2020).

Campagna, D., da Silva, A., and Braganholo, V. (2020). Achieving gdpr compliance through provenance: An extended model. In Anais do XXXV Simpósio Brasileiro de Bancos de Dados, pages 13–24, Porto Alegre, RS, Brasil. SBC.

Canon, R. S. (2020). The role of containers in reproducibility. In 2020 2nd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC), pages 19–25. IEEE.

Chen, X., Irshad, H., Chen, Y., Gehani, A., and Yegneswaran, V. (2021). CLARION: Sound and clear provenance tracking for microservice deployments. In 30th USENIX Security, pages 3989–4006.

Costa, F., Silva, V., de Oliveira, D., Ocaña, K. A. C. S., Ogasawara, E. S., Dias, J., and Mattoso, M. (2013). Capturing and querying workflow runtime provenance with PROV: a practical approach. In Guerrini, G., editor, EDBT/ICDT ’13, pages 282–289.

Datta, P., Polinsky, I., Inam, M. A., Bates, A., and Enck, W. (2022). ALASTOR: Reconstructing the provenance of serverless intrusions. In 31st USENIX Security, pages 2443–2460.

de Oliveira, D. C. M., Liu, J., and Pacitti, E. (2019). Data-Intensive Workflow Management: For Clouds and Data-Intensive and Scalable Computing Environments. Synthesis Lectures on Data Management. Morgan & Claypool Publishers.

Freire, J., Koop, D., Santos, E., and Silva, C. T. (2008). Provenance for computational tasks: A survey. Computing in Science & Engineering, 10(3):11–21.

Freitas, R. S., Barbosa, C. H., Guerra, G. M., Coutinho, A. L., and Rochinha, F. A. (2021). An encoder-decoder deep surrogate for reverse time migration in seismic imaging under uncertainty. Computational Geosciences, 25:1229–1250.

Gruening, B., Sallou, O., Moreno, P., da Veiga Leprevost, F., Ménager, H., Søndergaard, D., Röst, H., Sachsenberg, T., O’Connor, B., Madeira, F., Dominguez Del Angel, V., Crusoe, M. R., Varma, S., Blankenberg, D., Jimenez, R. C., BioContainers Community, and Perez-Riverol, Y. (2018). Recommendations for the packaging and containerizing of bioinformatics software. F1000Res, 7.

Han, R., Zheng, M., Byna, S., Tang, H., Dong, B., Dai, D., Chen, Y., Kim, D., Hassoun, J., and Thorsley, D. (2024). PROV-IO+: A cross-platform provenance framework for scientific data on hpc systems. IEEE Transactions on Parallel and Distributed Systems.

Kunstmann, L., Pina, D., de Oliveira, D., and Mattoso, M. (2024). ProvDeploy: Provenance-oriented containerization of high performance computing scientific workflows. arXiv preprint arXiv:2403.15324 / Under Review.

Kunstmann, L., Pina, D., de Oliveira, L. S., de Oliveira, D., and Mattoso, M. (2022). ProvDeploy: Explorando alternativas de conteinerização com proveniência para aplicações científicas com pad. In Anais do XXIII Simpósio em Sistemas Computacionais de Alto Desempenho, pages 49–60. SBC.

Lampa, S., Dahlö, M., Alvarsson, J., and Spjuth, O. (2019). Scipipe: A workflow library for agile development of complex and dynamic bioinformatics pipelines. GigaScience, 8(5):giz044.

Merkel, D. et al. (2014). Docker: lightweight linux containers for consistent development and deployment. Linux j, 239(2):2.

Modi, A., Reyad, M., Malik, T., and Gehani, A. (2023). Querying container provenance. In Companion Proceedings of the ACM Web Conference 2023, pages 1564–1567.

Moreau, L. and Groth, P. (2013). Provenance: an introduction to prov. Synthesis lectures on the semantic web: theory and technology, 3(4):1–129.

Murta, L., Braganholo, V., Chirigati, F., Koop, D., and Freire, J. (2015). noworkflow: capturing and analyzing provenance of scripts. In IPAW 2014, pages 71–83. Springer.

Novella, J. A., Emami Khoonsari, P., et al. (2019). Container-based bioinformatics with pachyderm. Bioinformatics, 35(5):839–846.

Olaya, P., Kennedy, D., et al. (2022). Building trust in earth science findings through data traceability and results explainability. IEEE TPDS, 34(2):704–717.

Orzechowski, M., Balis, B., Pawlik, K., Pawlik, M., and Malawski, M. (2018). Transparent deployment of scientific workflows across clouds-kubernetes approach. In 2018 IEEE/ACM UCC Companion, pages 9–10. IEEE.

Paranhos, R., Lage, M., and de Oliveira, D. (2023). Uso de grafos de proveniência para análise temporal de uso do solo em centros urbanos: uma abordagem prática. In Anais do XXXVIII Simpósio Brasileiro de Bancos de Dados, pages 457–462, Porto Alegre, RS, Brasil. SBC.

Pina, D., Chapman, A., Kunstmann, L., de Oliveira, D., and Mattoso, M. (2024). DLProv: A data-centric support for deep learning workflow analyses. In Companion of the 2024 ACM SIGMOD/PODS, Proceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning., DEEM ’24, pages 77–85. ACM.

Priedhorsky, R., Canon, R. S., Randles, T., and Younge, A. J. (2021). Minimizing privilege for building hpc containers. In IEEE/ACM SC, pages 1–14.

Satapathy, U., Thakur, R., Chattopadhyay, S., and Chakraborty, S. (2023). Disprotrack: Distributed provenance tracking over serverless applications. In IEEE INFOCOM 2023-IEEE Conference on Computer Communications, pages 1–10. IEEE.

Schlegel, M. and Sattler, K.-U. (2023). Management of machine learning lifecycle artifacts: A survey. SIGMOD Rec., 51(4):18–35.

Shaffer, T., Phung, T. S., Chard, K., and Thain, D. (2023). Landlord: Coordinating dynamic software environments to reduce container sprawl. IEEE Transactions on Parallel and Distributed Systems, 34(5):1376–1389.

Silva, V., Campos, V., Guedes, T., Camata, J., de Oliveira, D., Coutinho, A. L., Valduriez, P., and Mattoso, M. (2020). Dfanalyzer: runtime dataflow analysis tool for computational science and engineering applications. SoftwareX, 12:100592.

Straesser, M., Bauer, A., Leppich, R., Herbst, N., Chard, K., Foster, I., and Kounev, S. (2023). An empirical study of container image configurations and their impact on start times. In 2023 IEEE/ACM 23rd CCGrid, pages 94–105. IEEE.

Wofford, Q., Hurd, J., Greenberg, H., Bridges, P. G., and Ahrens, J. (2022). Complete provenance for application experiments with containers and hardware interface metadata. In 2022 IEEE/ACM CANOPIE-HPC, pages 12–24. IEEE.

Zhu, Y. and Zabaras, N. (2018). Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415–447.