Evaluating Reasoning Strategies for LLM-Based Repair of Solidity Smart Contracts

• William Valther Silva Martins PUC-Rio
• Daniel de Sousa Moraes PUC-Rio
• Eduardo Sardenberg Tavares PUC-Rio
• Antonio J. G. Busson BTG Pactual
• Julio Cesar Duarte IME
• Sergio Colcher PUC-Rio

DOI: https://doi.org/10.5753/webmedia_estendido.2026.34992

Resumo

Vulnerabilities in Solidity smart contracts can cause irreversible financial losses, making automated and reliable repair an important research challenge. Large Language Models (LLMs) have increasingly been explored for this task, yet existing approaches typically adopt a single reasoning strategy, such as Chain-of-Thought (CoT), either as a fixed prompting template or as a fallback for more challenging cases. Despite evidence that reasoning strategies can influence vulnerability detection and classification, their impact on automated repair remains unexplored. This paper evaluates whether the reasoning strategy affects repair effectiveness. We assess and compare four LLMs under zero-shot, Chain-of-Thought (CoT), and Tree-of-Thought (ToT) prompting on 200 vulnerable contracts from DAppSCAN, validating each repair for compilation, persistence of the vulnerability under an independent detector, and preservation of the contract interface. Across 987 repair attempts, the ToT prompt degraded repair effectiveness for every model evaluated, while plain zero-shot prompting was the strongest strategy for three of the four, inverting the pattern reported for detection. We trace the drop to the instruction to minimize behavioral change that our ToT prompt carries, rather than to structured exploration itself: the effect concentrates on the weaknesses whose repair requires a large edit. The verifiable subset is dominated by compiler-version weaknesses (SWC-102 and SWC-103), so the absolute rates characterize that class rather than vulnerability repair in general.

Palavras-chave: Smart Contracts, Large Language Models, Program Repair, Solidity, Prompt Engineering, Zero-Shot Reasoning

Referências

Sofia Bobadilla, Monica Jin, and Martin Monperrus. 2025. Do Automated Fixes Truly Mitigate Smart Contract Exploits? arXiv preprint arXiv:2501.04600 (2025).

Chong Chen et al. 2025. When ChatGPT meets smart contract vulnerability detection: How far are we? ACM Transactions on Software Engineering and Methodology 34, 4 (2025).

Lucas Bastos Germano and Julio Cesar Duarte. 2025. Repairing DeFi Vulnerabilities: Benchmarking LLMs with Executable Solidity Exploits. In Anais Estendidos do XXXI Simpósio Brasileiro de Sistemas Multimídia e Web (BrWeb3’2025). SBC, Porto Alegre, RS, Brasil. [link]

Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2012. GenProg: A Generic Method for Automatic Software Repair. In IEEE Transactions on Software Engineering, Vol. 38. 54–72.

Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019. Automated Program Repair. Commun. ACM 62, 12 (2019), 56–65.

InPlusLab. 2023. DAppSCAN: Vulnerability Dataset for Smart Contracts. [link].

Abhinav Jain et al. 2023. Two Timin’: Repairing Smart Contracts With A Two-Layered Approach. arXiv preprint arXiv:2401.01340 (2023).

Rabimba Karanjai, Sam Blackshear, Lei Xu, and Weidong Shi. 2025. SMARTIFY: A Multi-Agent Framework for Automated Vulnerability Detection and Repair in Solidity and Move Smart Contracts. arXiv preprint arXiv:2502.18515 (2025).

Rasoul Kiani and Victor S. Sheng. 2024. Automated Repair of Smart Contract Vulnerabilities: A Systematic Literature Review. Electronics 13, 19 (2024), 3942. [link]

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-shot Reasoners. Advances in Neural Information Processing Systems 35 (2022), 22199–22213.

Xiao Li et al. 2026. SCPatcher: Automated Smart Contract Code Repair via Retrieval-Augmented Generation and Knowledge Graph. arXiv preprint arXiv:2604.00687 (2026).

Jieyi Long. 2023. Large Language Model Guided Tree-of-Thought. arXiv preprint arXiv:2305.08291 (2023).

Gabriele Napoli and Valentina Gatteschi. 2023. Can ChatGPT support Solidity smart contracts security?. In Proceedings of the IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC).

Tai D. Nguyen et al. 2024. sGuard+: Machine Learning Guided Rule-Based Automated Vulnerability Repair on Smart Contracts. ACM Transactions on Software Engineering and Methodology (2024).

Tai D. Nguyen, Long H. Pham, and Jun Sun. 2021. SGUARD: Towards Fixing Vulnerable Smart Contracts Automatically. In Proceedings of the IEEE Symposium on Security and Privacy (S&P). 1215–1229.

Michael Rodler, Wenting Li, Ghassan O. Karame, and Lucas Davi. 2021. EVMPatch: Timely and Automated Patching of Ethereum Smart Contracts. In Proceedings of the 30th USENIX Security Symposium. 1289–1306.

Eduardo Sardenberg Tavares, Antonio José G. Busson, and Sérgio Colcher. 2025. Evaluating Zero-shot Reasoning with Agentic LLMs for Smart Contract Vulnerability Detection. In Proceedings of the XXXI Brazilian Symposium on Multimedia and the Web (WebMedia ’25). SBC, Porto Alegre, RS, Brasil, 306–310.

Che Wang, Jiashuo Zhang, Jianbo Gao, Libin Xia, Zhi Guan, and Zhong Chen. 2024. ContractTinker: LLM-Empowered Vulnerability Repair for Real-World Smart Contracts. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE ’24). arXiv:2409.09661.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837.

ZeKe Xiao, Qin Wang, Hammond Pearce, and Shiping Chen. 2025. Logic meets magic: LLMs cracking smart contract vulnerabilities. arXiv preprint arXiv:2501.07058 (2025).

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Advances in Neural Information Processing Systems 36 (2023).

Xiao Yu et al. 2020. SmartFix: Fixing Vulnerable Smart Contracts by Accelerated Deep Reinforcement Learning. In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE).

Zhuo Zhang et al. 2025. ACFix: Guiding LLMs to Fix Vulnerabilities by Context-awareness of Access Control. IEEE Transactions on Software Engineering (2025).