Preliminary Results of An Experiment on Leveraging Large Language Models to Assist Modelers in Interpreting DEVS Natural Language Models
Resumo
Discrete Event System Specification (DEVS) Natural Language (DNL) implements the DEVS simulation formalism using a natural languagelike notation. However, DNL models can still be complex, involving multiple inputs/outputs, internal/external state transitions, and arbitrary Java code blocks, which steepens the learning curve and reduces the efficiency of junior modelers. Concurrently, Large Language Models (LLMs) like ChatGPT have gained popularity across various domains for their ability to answer specific questions about referenced content. If an LLM tool could reference simulation models written in DNL, it could potentially greatly increase modeler efficiency. To this end, we developed GEM DEVS Chat, a tool designed to assist developers in understanding DEVS models within a simulation project. This paper presents GEM DEVS Chat and reports on an experiment conducted during a Modeling and Simulation course for undergraduate and graduate students. The experiment involved eight students, divided into control and experimental groups. Results indicate that students assisted by the tool understood DEVS models more quickly and accurately.Referências
Abbasiantaeb, Z., Yuan, Y., Kanoulas, E., and Aliannejadi, M. (2024). Let the LLMs talk: Simulating human-to-human conversational qa via zero-shot LLM-to-LLM interactions. In Proc. of 17th ACM WSDM, pages 8–17.
Agrawal, A., Kedia, N., Mohan, J., Panwar, A., Kwatra, N., Gulavani, B., Ramjee, R., and Tumanov, A. (2024). Vidur: A large-scale simulation framework for llm inference. Proceedings of Machine Learning and Systems, 6:351–366.
Alshareef, A., Keller, N., Carbo, P., and Zeigler, B. P. (2024). Generative AI with Modeling and Simulation of Activity and Flow-Based Diagrams. In Guisado-Lizar, J.-L., Riscos-Núñez, A., Morón-Fernández, M.-J., and Wainer, G., editors, Simulation Tools and Techniques, pages 95–109, Cham. Springer Nature Switzerland.
Alshareef, A. and Zeigler, B. P. (2020). Integration of activity specification into devs modeling & simulation development environment. In MSSiS, pages 46–55. SBC.
Araujo, R., Fornazin, M., and Pimentel, M. (2017). An analysis of the production of scientific knowledge in research published in the first 10 years of isys (2008-2017). iSys-Brazilian Journal of Information Systems, Porto Alegre, 10(4):45–65.
Basili, V. R. (1993). Applying the goal/question/metric paradigm in the experience factory. Software quality assurance and measurement: A worldwide perspective, 7(4):21–44.
Blas, M. J., Leone, H. P., and Gonnet, S. M. (2020). Building devs models from the functional design of software architecture components to estimate quality. In II MSSiS, pages 36–45. SBC.
Bulcão-Neto, R., Teixeira, P., Lebtag, B., Graciano-Neto, V., Macedo, A., and Zeigler, B. (2022). Simulation of IoT-oriented Fall Detection Systems Architectures for In-home Patients. IEEE Latin America Transactions, 21(1):16–26.
de Albuquerque, B. V. L., da Cunha, A. F. S., Souza, L., Siqueira, S. W. M., and Santos, R. (2024). Generating and reviewing programming codes with large language models: A systematic mapping study. In Proc. of the 20th SBSI, pages 70:1–70:10, Juiz de Fora, Brazil. ACM.
de França, B. B. N. and Graciano Neto, V. V. (2021). Opportunities for simulation in software engineering. In Anais do III Workshop em Modelagem e Simulação de Sistemas Intensivos em Software, pages 50–54. SBC.
de Oliveira, G. D., Porto, P. P. G., Alves, C. d. M. A., and Ralha, C. G. (2021). An agent-based model for simulating irrigated agriculture in the samambaia basin in goiás. Revista de Informática Teórica e Aplicada, 28(2):107–123.
Gao, C., Lan, X., Lu, Z., Mao, J., Piao, J., Wang, H., Jin, D., and Li, Y. (2023). S 3̂: Social-network simulation system with large language model-empowered agents. arXiv preprint arXiv:2307.14984.
Graciano Neto, V. V. and Kassab, M. (2023). What Every Engineer Should Know About Smart Cities. CRC Press - Taylor & Francis. 1st Edition. 254 p.
Jiang, C., Xu, H., Dong, M., Chen, J., Ye, W., Yan, M., Ye, Q., Zhang, J., Huang, F., and Zhang, S. (2024). Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27036–27046.
Kitchenham, B. and Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-001, Keele University and Durham University Joint Report.
Monteiro, E. L. and Maciel, R. S. P. (2020). Maturity models architecture: A large systematic mapping. iSys - Brazilian Journal of Information Systems, 13(2):110–140.
Moya, C. R. (2021). Como escolher o teste estatístico: um guia para o pesquisador iniciante. Câmara Brasileira do Livro, São Paulo, Brazil.
OliveiraJr, E., Cordeiro, A. F. R., and Nascimento, D. (2022). Surveying the open science knowledge in a southern brazilian university. In OpenSym 2022, pages 1:1–1:10, Madrid, Spain. ACM.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2012). Experimentation in software engineering. Springer Science & Business Media.
Zeigler, B. P., Sarjoughian, H. S., Duboz, R., and Souli, J.-C. (2012). Guide to Modeling and Simulation of Systems of Systems. Springer.
Agrawal, A., Kedia, N., Mohan, J., Panwar, A., Kwatra, N., Gulavani, B., Ramjee, R., and Tumanov, A. (2024). Vidur: A large-scale simulation framework for llm inference. Proceedings of Machine Learning and Systems, 6:351–366.
Alshareef, A., Keller, N., Carbo, P., and Zeigler, B. P. (2024). Generative AI with Modeling and Simulation of Activity and Flow-Based Diagrams. In Guisado-Lizar, J.-L., Riscos-Núñez, A., Morón-Fernández, M.-J., and Wainer, G., editors, Simulation Tools and Techniques, pages 95–109, Cham. Springer Nature Switzerland.
Alshareef, A. and Zeigler, B. P. (2020). Integration of activity specification into devs modeling & simulation development environment. In MSSiS, pages 46–55. SBC.
Araujo, R., Fornazin, M., and Pimentel, M. (2017). An analysis of the production of scientific knowledge in research published in the first 10 years of isys (2008-2017). iSys-Brazilian Journal of Information Systems, Porto Alegre, 10(4):45–65.
Basili, V. R. (1993). Applying the goal/question/metric paradigm in the experience factory. Software quality assurance and measurement: A worldwide perspective, 7(4):21–44.
Blas, M. J., Leone, H. P., and Gonnet, S. M. (2020). Building devs models from the functional design of software architecture components to estimate quality. In II MSSiS, pages 36–45. SBC.
Bulcão-Neto, R., Teixeira, P., Lebtag, B., Graciano-Neto, V., Macedo, A., and Zeigler, B. (2022). Simulation of IoT-oriented Fall Detection Systems Architectures for In-home Patients. IEEE Latin America Transactions, 21(1):16–26.
de Albuquerque, B. V. L., da Cunha, A. F. S., Souza, L., Siqueira, S. W. M., and Santos, R. (2024). Generating and reviewing programming codes with large language models: A systematic mapping study. In Proc. of the 20th SBSI, pages 70:1–70:10, Juiz de Fora, Brazil. ACM.
de França, B. B. N. and Graciano Neto, V. V. (2021). Opportunities for simulation in software engineering. In Anais do III Workshop em Modelagem e Simulação de Sistemas Intensivos em Software, pages 50–54. SBC.
de Oliveira, G. D., Porto, P. P. G., Alves, C. d. M. A., and Ralha, C. G. (2021). An agent-based model for simulating irrigated agriculture in the samambaia basin in goiás. Revista de Informática Teórica e Aplicada, 28(2):107–123.
Gao, C., Lan, X., Lu, Z., Mao, J., Piao, J., Wang, H., Jin, D., and Li, Y. (2023). S 3̂: Social-network simulation system with large language model-empowered agents. arXiv preprint arXiv:2307.14984.
Graciano Neto, V. V. and Kassab, M. (2023). What Every Engineer Should Know About Smart Cities. CRC Press - Taylor & Francis. 1st Edition. 254 p.
Jiang, C., Xu, H., Dong, M., Chen, J., Ye, W., Yan, M., Ye, Q., Zhang, J., Huang, F., and Zhang, S. (2024). Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27036–27046.
Kitchenham, B. and Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-001, Keele University and Durham University Joint Report.
Monteiro, E. L. and Maciel, R. S. P. (2020). Maturity models architecture: A large systematic mapping. iSys - Brazilian Journal of Information Systems, 13(2):110–140.
Moya, C. R. (2021). Como escolher o teste estatístico: um guia para o pesquisador iniciante. Câmara Brasileira do Livro, São Paulo, Brazil.
OliveiraJr, E., Cordeiro, A. F. R., and Nascimento, D. (2022). Surveying the open science knowledge in a southern brazilian university. In OpenSym 2022, pages 1:1–1:10, Madrid, Spain. ACM.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2012). Experimentation in software engineering. Springer Science & Business Media.
Zeigler, B. P., Sarjoughian, H. S., Duboz, R., and Souli, J.-C. (2012). Guide to Modeling and Simulation of Systems of Systems. Springer.
Publicado
30/09/2024
Como Citar
GRACIANO NETO, Valdemar Vicente; KELLER, Nicholas; KIM, Doohwan DH; SEO, Chungman; CARBO, Priscilla; ZEIGLER, Bernard.
Preliminary Results of An Experiment on Leveraging Large Language Models to Assist Modelers in Interpreting DEVS Natural Language Models. In: WORKSHOP EM MODELAGEM E SIMULAÇÃO DE SISTEMAS INTENSIVOS EM SOFTWARE (MSSIS), 6. , 2024, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 41-50.
DOI: https://doi.org/10.5753/mssis.2024.3714.
