Can Large Language Models Engineer Parallel Workflows?
Resumo
Scientific workflows are specified using specialized languages to define parallel execution, an expertise that domain users may lack. Large Language Models (LLMs) can lower this barrier, but existing evaluations stop at syntactic analysis and do not verify whether the generated specifications execute or preserve parallelism. In this paper, we evaluated four LLMs in two workflow engineering tasks (workflow specification from natural-language and conversion across workflow systems), covering three workflow systems, three workflows, and two prompting strategies. For each generated workflow specification, we analyzed its executability and whether it executes in parallel. Results show that lexical similarity to expert-developed workflows is a poor predictor of executability, that in-context examples improve execution rates unevenly across systems, and that several workflow specifications collapse into serial execution. Even those that support parallelism are consistently slower than the expert-developed workflow.
Referências
Alam, K. and Roy, B. (2025). From prompt to pipeline: Large language models for scientific workflow development in bioinformatics. In arXiv, arXiv. arXiv.
Amstutz, P., Crusoe, M. R., Tijanić, N., Chapman, B., Chilton, J., Heuer, M., Kartashov, A., Leehr, D., Ménager, H., Nedeljkovich, M., et al. (2016). Common workflow language, draft-3. Figshare.
Babuji, Y., Woodard, A., Li, Z., Katz, D. S., Clifford, B., Kumar, R., Lacinski, L., Chard, R., Wozniak, J. M., Foster, I., Wilde, M., and Chard, K. (2019). Parsl: Pervasive parallel programming in python. In Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’19), pages 25–36. ACM.
Badia, R. M., Ayguade, E., and Labarta, J. (2017). Workflows for science: A challenge when facing the convergence of hpc and big data. Supercomputing frontiers and innovations, 4(1):27–47.
Badia, R. M., Conejero, J., Ejarque, J., Lezzi, D., Lordan, F., and Sirvent, R. (2026). Programming task-based workflows with COMPSs. In Workflow Systems for Large-Scale Scientific Data Analysis. Berlin Universities Publishing.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. (2021). Evaluating large language models trained on code.
Cohere (2026). Command a+. [link]. [Accessed 16-08-2026].
Da Silva, R. F., Bard, D., Chard, K., De Witt, S., Foster, I. T., Gibbs, T., Goble, C., Godoy, W., Gustafsson, J., Haus, U.-U., et al. (2024). Workflows community summit 2024: Future trends and challenges in scientific workflows. arXiv preprint arXiv:2410.14943.
de Oliveira, D., Ogasawara, E. S., Baião, F. A., and Mattoso, M. (2010). Scicumulus: A lightweight cloud middleware to explore many task computing paradigm in scientific workflows. In IEEE International Conference on Cloud Computing, CLOUD 2010, Miami, FL, USA, 5-10 July, 2010, pages 378–385. IEEE Computer Society.
de Oliveira, D. C. M., Liu, J., and Pacitti, E. (2019). Data-Intensive Workflow Management: For Clouds and Data-Intensive and Scalable Computing Environments. Synthesis Lectures on Data Management. Morgan & Claypool Publishers.
Deelman, E., Vahi, K., Juve, G., Rynge, M., Callaghan, S., Maechling, P. J., Mayani, R., Chen, W., Ferreira da Silva, R., Livny, M., and Wenger, K. (2015). Pegasus, a workflow management system for science automation. Future Gener. Comput. Syst., 46(C):17–35.
Deepseek (2026). Deepseek v4. [link]. [Accessed 16-08-2026].
Di Tommaso, P., Chatzou, M., Floden, E. W., Barja, P. P., Palumbo, E., and Notredame, C. (2017). Nextflow enables reproducible computational workflows. Nature Biotechnology, 35(4):316–319.
Duque, A., Syed, A., Day, K. V., Berry, M. J., Katz, D. S., and Kindratenko, V. V. (2023). Leveraging large language models to build and execute computational workflows. arXiv preprint arXiv:2312.07711.
Google (2026). Gemma 4. [link]. [Accessed 16-08-2026].
Gu, Y., Cao, J., Guo, Y., Qian, S., and Guan, W. (2023). Plan, generate and match: Scientific workflow recommendation with large language models. In Lecture Notes in Computer Science, Lecture Notes in Computer Science, pages 86–102. Springer Nature Switzerland, Cham.
Liu, J., Pacitti, E., Valduriez, P., and Mattoso, M. (2015). A survey of data-intensive scientific workflow management. Journal of Grid Computing, 13(4):457–493.
Ljaljevic, S., Jorba, J., and Iserte, S. (2026). Exploring the role of large language models in high-performance computing programming: A survey. Future Generation Computer Systems, 184:108618.
Miyashita, Y., Tung, P. K. M., and Barthélemy, J. (2024). Llm as hpc expert: Extending rag architecture for hpc data.
OpenAI (2026). Gpt-5.5. [link]. [Accessed 16-08-2026].
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Isabelle, P., Charniak, E., and Lin, D., editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Popović, M. (2015). chrF: character n-gram F-score for automatic MT evaluation. In Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Hokamp, C., Huck, M., Logacheva, V., and Pecina, P., editors, Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., and Chadha, A. (2024). A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927.
Sänger, M., De Mecquenem, N., Lewińska, K. E., Bountris, V., Lehmann, F., Leser, U., and Kosch, T. (2024). A qualitative assessment of using chatgpt as large language model for scientific workflow development. GigaScience, 13:giae030.
Yildiz, O. and Peterka, T. (2025). Do large language models speak scientific workflows? In Proceedings of the SC ’25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC Workshops ’25, page 2225–2233, New York, NY, USA. Association for Computing Machinery.
Yin, J., Hines, J., Herron, E., Ghosal, T., Liu, H., Prentice, S., Lama, V., and Wang, F. (2025). chatHPC: Empowering HPC users with large language models. J. Supercomput., 81(1).
Zhao, W. X. et al. (2023). A survey of large language models. arXiv:2303.18223, 1(2).
Amstutz, P., Crusoe, M. R., Tijanić, N., Chapman, B., Chilton, J., Heuer, M., Kartashov, A., Leehr, D., Ménager, H., Nedeljkovich, M., et al. (2016). Common workflow language, draft-3. Figshare.
Babuji, Y., Woodard, A., Li, Z., Katz, D. S., Clifford, B., Kumar, R., Lacinski, L., Chard, R., Wozniak, J. M., Foster, I., Wilde, M., and Chard, K. (2019). Parsl: Pervasive parallel programming in python. In Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’19), pages 25–36. ACM.
Badia, R. M., Ayguade, E., and Labarta, J. (2017). Workflows for science: A challenge when facing the convergence of hpc and big data. Supercomputing frontiers and innovations, 4(1):27–47.
Badia, R. M., Conejero, J., Ejarque, J., Lezzi, D., Lordan, F., and Sirvent, R. (2026). Programming task-based workflows with COMPSs. In Workflow Systems for Large-Scale Scientific Data Analysis. Berlin Universities Publishing.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. (2021). Evaluating large language models trained on code.
Cohere (2026). Command a+. [link]. [Accessed 16-08-2026].
Da Silva, R. F., Bard, D., Chard, K., De Witt, S., Foster, I. T., Gibbs, T., Goble, C., Godoy, W., Gustafsson, J., Haus, U.-U., et al. (2024). Workflows community summit 2024: Future trends and challenges in scientific workflows. arXiv preprint arXiv:2410.14943.
de Oliveira, D., Ogasawara, E. S., Baião, F. A., and Mattoso, M. (2010). Scicumulus: A lightweight cloud middleware to explore many task computing paradigm in scientific workflows. In IEEE International Conference on Cloud Computing, CLOUD 2010, Miami, FL, USA, 5-10 July, 2010, pages 378–385. IEEE Computer Society.
de Oliveira, D. C. M., Liu, J., and Pacitti, E. (2019). Data-Intensive Workflow Management: For Clouds and Data-Intensive and Scalable Computing Environments. Synthesis Lectures on Data Management. Morgan & Claypool Publishers.
Deelman, E., Vahi, K., Juve, G., Rynge, M., Callaghan, S., Maechling, P. J., Mayani, R., Chen, W., Ferreira da Silva, R., Livny, M., and Wenger, K. (2015). Pegasus, a workflow management system for science automation. Future Gener. Comput. Syst., 46(C):17–35.
Deepseek (2026). Deepseek v4. [link]. [Accessed 16-08-2026].
Di Tommaso, P., Chatzou, M., Floden, E. W., Barja, P. P., Palumbo, E., and Notredame, C. (2017). Nextflow enables reproducible computational workflows. Nature Biotechnology, 35(4):316–319.
Duque, A., Syed, A., Day, K. V., Berry, M. J., Katz, D. S., and Kindratenko, V. V. (2023). Leveraging large language models to build and execute computational workflows. arXiv preprint arXiv:2312.07711.
Google (2026). Gemma 4. [link]. [Accessed 16-08-2026].
Gu, Y., Cao, J., Guo, Y., Qian, S., and Guan, W. (2023). Plan, generate and match: Scientific workflow recommendation with large language models. In Lecture Notes in Computer Science, Lecture Notes in Computer Science, pages 86–102. Springer Nature Switzerland, Cham.
Liu, J., Pacitti, E., Valduriez, P., and Mattoso, M. (2015). A survey of data-intensive scientific workflow management. Journal of Grid Computing, 13(4):457–493.
Ljaljevic, S., Jorba, J., and Iserte, S. (2026). Exploring the role of large language models in high-performance computing programming: A survey. Future Generation Computer Systems, 184:108618.
Miyashita, Y., Tung, P. K. M., and Barthélemy, J. (2024). Llm as hpc expert: Extending rag architecture for hpc data.
OpenAI (2026). Gpt-5.5. [link]. [Accessed 16-08-2026].
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Isabelle, P., Charniak, E., and Lin, D., editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Popović, M. (2015). chrF: character n-gram F-score for automatic MT evaluation. In Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Hokamp, C., Huck, M., Logacheva, V., and Pecina, P., editors, Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., and Chadha, A. (2024). A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927.
Sänger, M., De Mecquenem, N., Lewińska, K. E., Bountris, V., Lehmann, F., Leser, U., and Kosch, T. (2024). A qualitative assessment of using chatgpt as large language model for scientific workflow development. GigaScience, 13:giae030.
Yildiz, O. and Peterka, T. (2025). Do large language models speak scientific workflows? In Proceedings of the SC ’25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC Workshops ’25, page 2225–2233, New York, NY, USA. Association for Computing Machinery.
Yin, J., Hines, J., Herron, E., Ghosal, T., Liu, H., Prentice, S., Lama, V., and Wang, F. (2025). chatHPC: Empowering HPC users with large language models. J. Supercomput., 81(1).
Zhao, W. X. et al. (2023). A survey of large language models. arXiv:2303.18223, 1(2).
Publicado
03/11/2026
Como Citar
WOYAMES, Paula; KUNSTMANN, Liliane; PINA, Débora; MATTOSO, Marta; OLIVEIRA, Daniel de.
Can Large Language Models Engineer Parallel Workflows?. In: SIMPÓSIO EM SISTEMAS COMPUTACIONAIS DE ALTO DESEMPENHO (SSCAD), 27. , 2026, Natal/RN.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 287-300.
DOI: https://doi.org/10.5753/sscad.2026.34933.
