Comparative Analysis of Smart Contract Generation Using Large Language Models
Resumo
In recent years, blockchain technology has established itself as an effective, secure, and transparent data storage solution. In this context, smart contracts play a fundamental role by enabling the automated execution of agreements without intermediaries. With the advancement of language models, the opportunity to automatically generate these contracts has emerged, raising concerns about their reliability and potential vulnerabilities. This article proposes a comparative analysis of the available language models for developing smart contracts using Ethereum Virtual Machine’s contracts as a case study. Experiments were made using various Large language models using different metrics to evaluate the susceptibility to vulnerabilities and computational cost. After comparing various models, ChatGPT appears to be the most suitable for generating smart contracts due to its higher compilation rate and, consequently, a larger sample size, despite detecting more vulnerabilities.
Palavras-chave:
Smart Contract, Blockchain, Large Language Models, Comparative Analysis
Referências
Ahmed, S. U., Danish, A., Ahmad, N., and Ahmad, T. (2024). Smart contract generation through NLP and blockchain for legal documents. Procedia Computer Science, 235:2529–2537. International Conference on Machine Learning and Data Engineering (ICMLDE 2023).
Bashir, I. (2020). Mastering Blockchain: A Deep Dive Into Distributed Ledgers, Consensus Protocols, Smart Contracts, DApps, Cryptocurrencies, Ethereum, and More. Expert Insight. Packt Publishing.
Boiko, D. A., MacKnight, R., and Gomes, G. (2023). Emergent autonomous scientific research capabilities of large language models.
Buterin, V. (2014). Ethereum white paper: A next-generation smart contract and decentralized application platform. Accessed: 2024-11-02.
Chen, B., Zhang, Z., Langrené, N., and Zhu, S. (2024). Unleashing the potential of prompt engineering in large language models: A comprehensive review.
Fadi, B., Napoli, E. A., Gatteschi, V., Schifanella, C., et al. (2024). Automatic smart contract generation through LLMs: When the stochastic parrot fails. In Proceedings of DLT 2024. CEUR.
Fan, Y., Chen, E., Zhu, Y., He, X., Yau, S. S., and Pandya, K. (2023). Automatic generation of smart contracts from real-world contracts in natural language. In 2023 IEEE Smart World Congress (SWC), pages 1–8.
Farokhnia, S. and Goharshady, A. K. (2023). Reducing the gas usage of Ethereum smart contracts without a sidechain. In 2023 IEEE International Conference on Blockchain and Cryptocurrency (ICBC), pages 1–3.
Feist, J., Grieco, G., and Groce, A. (2019). Slither: A static analysis framework for smart contracts. In 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB), pages 8–15.
Islam, S., Islam, M. J., Hossain, M., Noor, S., Kwak, K.-S., and Islam, S. M. R. (2023). A survey on consensus algorithms in blockchain-based applications: Architecture, taxonomy, and operational issues. IEEE Access, 11:39066–39082.
Jayakody, R. and Dias, G. (2024). Performance of recent large language models for a low-resourced language. In 2024 International Conference on Asian Language Processing (IALP), pages 162–167.
Jurgelaitis, M., Ceponiene, L., and Butkiene, R. (2022). Solidity code generation from UML state machines in model-driven smart contract development. IEEE Access, 10:33465–33481.
Lewis, D.-M., DeRenzi, B., Misomali, A., Nyirenda, T., Phiri, E., Chifisi, L., Makwenda, C., and Lesh, N. (2024). Human review for post-training improvement of low-resource language performance in large language models. In 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI), pages 592–597.
Li, C. (2021). Gas estimation and optimization for smart contracts on Ethereum. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1082–1086.
Ma, F., Fu, Y., Ren, M., Wang, M., Jiang, Y., Zhang, K., Li, H., and Shi, X. (2019). EVM: From offline detection to online reinforcement for Ethereum virtual machine. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 554–558. IEEE.
Muneeb, M., Raza, Z., Haq, I. U., and Shafiq, O. (2022). Smartcon: A blockchain-based framework for smart contracts and transaction management. IEEE Access, 10:23687–23699.
Nakamoto, S. (2008). Bitcoin: A peer-to-peer electronic cash system. Decentralized Business Review.
Napoli, E. A., Barbàra, F., Gatteschi, V., and Schifanella, C. (2024). Leveraging large language models for automatic smart contract generation. In 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), pages 701–710.
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2024). A comprehensive overview of large language models.
Peng, J., Han, Z., Zhang, H., Ye, J., Liu, C., Liu, B., Guo, M., Chen, H., Lin, Z., and Tang, Y. (2024). A multilingual text detoxification method based on few-shot learning and co-star framework. Working Notes of CLEF.
Reynolds, L. and McDonell, K. (2021). Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pages 1–7.
Science, G. D. and Division, S. G. A. (2023). Prompt engineering playbook. Online. Accessed: 2024-10-11.
Shen, Y. (2024). Enhancing English language education with ChatGPT. In Proceedings of the 2024 International Conference on Artificial Intelligence and Communication (ICAIC 2024), pages 512–521. Atlantis Press.
Shynkarenko, D. and Kopp, A. (2022). Towards the approach to building smart contracts based on business rules using natural language processing. Grail of Science, (22):144–150.
Sujeetha, R. and Deiva Preetha, C. A. S. (2021). A literature survey on smart contract testing and analysis for smart contract based blockchain application development. In 2021 2nd International Conference on Smart Electronics and Communication (ICOSEC), pages 378–385.
Szabo, N. (1997). The idea of smart contracts. Accessed: 2024-10-22.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
Wang, D., Shan, M., and Tong, N. (2024). Smart contract vulnerability detection based on machine learning. In 2024 6th International Conference on Electronic Engineering and Informatics (EEI), pages 1038–1042.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. (2022a). Emergent abilities of large language models.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022b). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
Wei, Z., Sun, J., Zhang, Z., Zhang, X., Li, M., and Zhu, L. (2023). A comparative evaluation of automated analysis tools for Solidity smart contracts.
Wu, G., Wang, H., Lai, X., Wang, M., He, D., and Chan, S. (2024). A comprehensive survey of smart contract security: State of the art and research directions. Journal of Network and Computer Applications, 226:103882.
Yang, H., Zhang, J., Gu, X., and Cui, Z. (2022). Smart contract vulnerability detection based on abstract syntax tree. In 2022 8th International Symposium on System Security, Safety, and Reliability (ISSSR), pages 169–170.
Zhao, B. (2023). Reverse real-time model detection of chain smart contract security based on association method. In 2023 International Conference on Telecommunications, Electronics and Informatics (ICTEI), pages 759–762.
Zhao, J., Chen, X., Yang, G., and Shen, Y. (2024). Automatic smart contract comment generation via large language models and in-context learning. Information and Software Technology, 168:107405.
Zheng, Z., Xie, S., Dai, H.-N., Chen, W., Chen, X., Weng, J., and Imran, M. (2020). An overview on smart contracts: Challenges, advances and platforms. Future Generation Computer Systems, 105:475–491.
Bashir, I. (2020). Mastering Blockchain: A Deep Dive Into Distributed Ledgers, Consensus Protocols, Smart Contracts, DApps, Cryptocurrencies, Ethereum, and More. Expert Insight. Packt Publishing.
Boiko, D. A., MacKnight, R., and Gomes, G. (2023). Emergent autonomous scientific research capabilities of large language models.
Buterin, V. (2014). Ethereum white paper: A next-generation smart contract and decentralized application platform. Accessed: 2024-11-02.
Chen, B., Zhang, Z., Langrené, N., and Zhu, S. (2024). Unleashing the potential of prompt engineering in large language models: A comprehensive review.
Fadi, B., Napoli, E. A., Gatteschi, V., Schifanella, C., et al. (2024). Automatic smart contract generation through LLMs: When the stochastic parrot fails. In Proceedings of DLT 2024. CEUR.
Fan, Y., Chen, E., Zhu, Y., He, X., Yau, S. S., and Pandya, K. (2023). Automatic generation of smart contracts from real-world contracts in natural language. In 2023 IEEE Smart World Congress (SWC), pages 1–8.
Farokhnia, S. and Goharshady, A. K. (2023). Reducing the gas usage of Ethereum smart contracts without a sidechain. In 2023 IEEE International Conference on Blockchain and Cryptocurrency (ICBC), pages 1–3.
Feist, J., Grieco, G., and Groce, A. (2019). Slither: A static analysis framework for smart contracts. In 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB), pages 8–15.
Islam, S., Islam, M. J., Hossain, M., Noor, S., Kwak, K.-S., and Islam, S. M. R. (2023). A survey on consensus algorithms in blockchain-based applications: Architecture, taxonomy, and operational issues. IEEE Access, 11:39066–39082.
Jayakody, R. and Dias, G. (2024). Performance of recent large language models for a low-resourced language. In 2024 International Conference on Asian Language Processing (IALP), pages 162–167.
Jurgelaitis, M., Ceponiene, L., and Butkiene, R. (2022). Solidity code generation from UML state machines in model-driven smart contract development. IEEE Access, 10:33465–33481.
Lewis, D.-M., DeRenzi, B., Misomali, A., Nyirenda, T., Phiri, E., Chifisi, L., Makwenda, C., and Lesh, N. (2024). Human review for post-training improvement of low-resource language performance in large language models. In 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI), pages 592–597.
Li, C. (2021). Gas estimation and optimization for smart contracts on Ethereum. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1082–1086.
Ma, F., Fu, Y., Ren, M., Wang, M., Jiang, Y., Zhang, K., Li, H., and Shi, X. (2019). EVM: From offline detection to online reinforcement for Ethereum virtual machine. In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 554–558. IEEE.
Muneeb, M., Raza, Z., Haq, I. U., and Shafiq, O. (2022). Smartcon: A blockchain-based framework for smart contracts and transaction management. IEEE Access, 10:23687–23699.
Nakamoto, S. (2008). Bitcoin: A peer-to-peer electronic cash system. Decentralized Business Review.
Napoli, E. A., Barbàra, F., Gatteschi, V., and Schifanella, C. (2024). Leveraging large language models for automatic smart contract generation. In 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC), pages 701–710.
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2024). A comprehensive overview of large language models.
Peng, J., Han, Z., Zhang, H., Ye, J., Liu, C., Liu, B., Guo, M., Chen, H., Lin, Z., and Tang, Y. (2024). A multilingual text detoxification method based on few-shot learning and co-star framework. Working Notes of CLEF.
Reynolds, L. and McDonell, K. (2021). Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pages 1–7.
Science, G. D. and Division, S. G. A. (2023). Prompt engineering playbook. Online. Accessed: 2024-10-11.
Shen, Y. (2024). Enhancing English language education with ChatGPT. In Proceedings of the 2024 International Conference on Artificial Intelligence and Communication (ICAIC 2024), pages 512–521. Atlantis Press.
Shynkarenko, D. and Kopp, A. (2022). Towards the approach to building smart contracts based on business rules using natural language processing. Grail of Science, (22):144–150.
Sujeetha, R. and Deiva Preetha, C. A. S. (2021). A literature survey on smart contract testing and analysis for smart contract based blockchain application development. In 2021 2nd International Conference on Smart Electronics and Communication (ICOSEC), pages 378–385.
Szabo, N. (1997). The idea of smart contracts. Accessed: 2024-10-22.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
Wang, D., Shan, M., and Tong, N. (2024). Smart contract vulnerability detection based on machine learning. In 2024 6th International Conference on Electronic Engineering and Informatics (EEI), pages 1038–1042.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. (2022a). Emergent abilities of large language models.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. (2022b). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
Wei, Z., Sun, J., Zhang, Z., Zhang, X., Li, M., and Zhu, L. (2023). A comparative evaluation of automated analysis tools for Solidity smart contracts.
Wu, G., Wang, H., Lai, X., Wang, M., He, D., and Chan, S. (2024). A comprehensive survey of smart contract security: State of the art and research directions. Journal of Network and Computer Applications, 226:103882.
Yang, H., Zhang, J., Gu, X., and Cui, Z. (2022). Smart contract vulnerability detection based on abstract syntax tree. In 2022 8th International Symposium on System Security, Safety, and Reliability (ISSSR), pages 169–170.
Zhao, B. (2023). Reverse real-time model detection of chain smart contract security based on association method. In 2023 International Conference on Telecommunications, Electronics and Informatics (ICTEI), pages 759–762.
Zhao, J., Chen, X., Yang, G., and Shen, Y. (2024). Automatic smart contract comment generation via large language models and in-context learning. Information and Software Technology, 168:107405.
Zheng, Z., Xie, S., Dai, H.-N., Chen, W., Chen, X., Weng, J., and Imran, M. (2020). An overview on smart contracts: Challenges, advances and platforms. Future Generation Computer Systems, 105:475–491.
Publicado
19/05/2025
Como Citar
BENEDITO DOS SANTOS, Hiago Vinícius; ROSA DOS SANTOS JANUARIO, Raissa; CARVALHO ZANATTA, Ravelly; NEVES MATOS, Saulo; UEYAMA, Jó.
Comparative Analysis of Smart Contract Generation Using Large Language Models. In: SIMPÓSIO BRASILEIRO DE REDES DE COMPUTADORES E SISTEMAS DISTRIBUÍDOS (SBRC), 43. , 2025, Natal/RN.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2025
.
p. 112-125.
ISSN 2177-9384.
DOI: https://doi.org/10.5753/sbrc.2025.5848.
