SolForge: A Large Language Model-Based Approach for the Differential Testing of Solidity Compilers
Resumo
Solidity is the primary language used to develop smart contracts on Ethereum. Because deployed contracts are immutable, compiler reliability is particularly important, as compilation defects can introduce persistent vulnerabilities into generated bytecode. Traditional testing approaches often struggle to explore the semantic complexity required to expose edge-case compiler failures. This paper presents SolForge, an automated fuzzing and differential testing framework driven by Large Language Models (LLMs). Rather than relying on handcrafted templates or model fine-tuning, SolForge uses combinatorial prompt generation to produce structurally diverse Solidity contracts. These contracts are then compiled across different compiler configurations, including solc (standard and optimized) and solang, to identify behavioral inconsistencies and internal failures. The experimental evaluation produced 1,357 syntactically valid contracts and revealed 1,041 compilation disagreements across compiler targets and configurations, most of which corresponded to architectural incompatibilities between EVM-oriented Solidity code and the Solana backend of solang. In addition, the approach exposed 29 internal compiler errors, including 9 unhandled exceptions in the official solc compiler. The results suggest that prompt-engineered LLMs can serve as a complementary strategy for differential compiler testing in blockchain environments.
Palavras-chave:
Solidity, Smart Contracts, Code Generation, Fuzzing, Differential Testing
Referências
Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A survey of compiler testing. ACM Computing Surveys (CSUR) 53, 1 (2020), 1–36.
Lantian Li, Zhihao Liu, and Zhongxing Yu. 2025. Solsmith: Solidity Random Program Generator for Compiler Testing. arXiv preprint arXiv:2506.03909 (2025).
Ruichao Liang, Jing Chen, CongWu, Kun He, YuemingWu, Ruochen Cao, Ruiying Du, Ziming Zhao, and Yang Liu. 2025. Vulseye: Detect smart contract vulnerabilities via stateful directed graybox fuzzing. IEEE Transactions on Information Forensics and Security (2025).
Haoyang Ma, Alastair F Donaldson, Qingchao Shen, Yongqiang Tian, Junjie Chen, and Shing-Chi Cheung. 2025. Bounded Exhaustive Random Program Generation for Testing Solidity Compilers and Analyzers. arXiv preprint arXiv:2503.20332 (2025).
Haoyang Ma, Wuqi Zhang, Qingchao Shen, Yongqiang Tian, Junjie Chen, and Shing-Chi Cheung. 2024. Towards Understanding the Bugs in Solidity Compiler. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ’24). ACM, 1312–1324. DOI: 10.1145/3650212.3680362
Jie Ma, Ningyu He, Jinwen Xi, Mingzhe Xing, Haoyu Wang, Ying Gao, and Yinliang Yue. 2025. OpDiffer: LLM-Assisted Opcode-Level Differential Testing of Ethereum Virtual Machine. arXiv preprint arXiv:2504.12034 (2025).
Valentin JM Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J Schwartz, and Maverick Woo. 2019. The art, science, and engineering of fuzzing: A survey. IEEE Transactions on Software Engineering 47, 11 (2019), 2312–2331.
William M McKeeman. 1998. Differential testing for software. Digital Technical Journal 10, 1 (1998), 100–107.
Charalambos Mitropoulos, Thodoris Sotiropoulos, Sotiris Ioannidis, and Dimitris Mitropoulos. 2023. Syntax-Aware Mutation for Testing the Solidity Compiler. In European Symposium on Research in Computer Security. Springer, 327–347.
Hui Peng, Yan Shoshitaishvili, and Mathias Payer. 2018. T-Fuzz: fuzzing by program transformation. In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 697–710.
Eugenia Politou, Fran Casino, Efthimios Alepis, and Constantinos Patsakis. 2019. Blockchain mutability: Challenges and proposed solutions. IEEE Transactions on Emerging Topics in Computing 9, 4 (2019), 1972–1986.
Nick Szabo. 1996. Smart contracts: building blocks for digital markets. Extropy 16, 2 (1996), 18.
Zhenzhou Tian, Fanfan Wang, Yanping Chen, and Lingwei Chen. 2024. Differential testing solidity compiler through deep contract manipulation and mutation. Software Quality Journal 32, 2 (2024), 765–790.
Gautami Tripathi, Mohd Abdul Ahad, and Gabriella Casalino. 2023. A comprehensive review of blockchain technology: Underlying principles and historical background with future challenges. Decision Analytics Journal 9 (2023), 100344.
GavinWood et al. 2014. Ethereum: A secure decentralised generalised transaction ledger. Ethereum project yellow paper 151, 2014 (2014), 1–32.
Qihao Zhu et al. 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence. arXiv preprint arXiv:2406.11931 (2024).
Lantian Li, Zhihao Liu, and Zhongxing Yu. 2025. Solsmith: Solidity Random Program Generator for Compiler Testing. arXiv preprint arXiv:2506.03909 (2025).
Ruichao Liang, Jing Chen, CongWu, Kun He, YuemingWu, Ruochen Cao, Ruiying Du, Ziming Zhao, and Yang Liu. 2025. Vulseye: Detect smart contract vulnerabilities via stateful directed graybox fuzzing. IEEE Transactions on Information Forensics and Security (2025).
Haoyang Ma, Alastair F Donaldson, Qingchao Shen, Yongqiang Tian, Junjie Chen, and Shing-Chi Cheung. 2025. Bounded Exhaustive Random Program Generation for Testing Solidity Compilers and Analyzers. arXiv preprint arXiv:2503.20332 (2025).
Haoyang Ma, Wuqi Zhang, Qingchao Shen, Yongqiang Tian, Junjie Chen, and Shing-Chi Cheung. 2024. Towards Understanding the Bugs in Solidity Compiler. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ’24). ACM, 1312–1324. DOI: 10.1145/3650212.3680362
Jie Ma, Ningyu He, Jinwen Xi, Mingzhe Xing, Haoyu Wang, Ying Gao, and Yinliang Yue. 2025. OpDiffer: LLM-Assisted Opcode-Level Differential Testing of Ethereum Virtual Machine. arXiv preprint arXiv:2504.12034 (2025).
Valentin JM Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J Schwartz, and Maverick Woo. 2019. The art, science, and engineering of fuzzing: A survey. IEEE Transactions on Software Engineering 47, 11 (2019), 2312–2331.
William M McKeeman. 1998. Differential testing for software. Digital Technical Journal 10, 1 (1998), 100–107.
Charalambos Mitropoulos, Thodoris Sotiropoulos, Sotiris Ioannidis, and Dimitris Mitropoulos. 2023. Syntax-Aware Mutation for Testing the Solidity Compiler. In European Symposium on Research in Computer Security. Springer, 327–347.
Hui Peng, Yan Shoshitaishvili, and Mathias Payer. 2018. T-Fuzz: fuzzing by program transformation. In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 697–710.
Eugenia Politou, Fran Casino, Efthimios Alepis, and Constantinos Patsakis. 2019. Blockchain mutability: Challenges and proposed solutions. IEEE Transactions on Emerging Topics in Computing 9, 4 (2019), 1972–1986.
Nick Szabo. 1996. Smart contracts: building blocks for digital markets. Extropy 16, 2 (1996), 18.
Zhenzhou Tian, Fanfan Wang, Yanping Chen, and Lingwei Chen. 2024. Differential testing solidity compiler through deep contract manipulation and mutation. Software Quality Journal 32, 2 (2024), 765–790.
Gautami Tripathi, Mohd Abdul Ahad, and Gabriella Casalino. 2023. A comprehensive review of blockchain technology: Underlying principles and historical background with future challenges. Decision Analytics Journal 9 (2023), 100344.
GavinWood et al. 2014. Ethereum: A secure decentralised generalised transaction ledger. Ethereum project yellow paper 151, 2014 (2014), 1–32.
Qihao Zhu et al. 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence. arXiv preprint arXiv:2406.11931 (2024).
Publicado
08/09/2026
Como Citar
SALVI, Gabriel Zortea; BRAGA, Andrei; DAL BIANCO, Guilherme; FEITOSA, Samuel.
SolForge: A Large Language Model-Based Approach for the Differential Testing of Solidity Compilers. In: SIMPÓSIO BRASILEIRO DE LINGUAGENS DE PROGRAMAÇÃO (SBLP), 30. , 2026, São Paulo/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 133-140.
DOI: https://doi.org/10.5753/sblp.2026.29638.
