Regression of the effectiveness of vulnerability analyzers in blockchain smart contracts

• Rafael Santa Rosa Alves UNICAMP
• Marco Amaral Henriques UNICAMP

DOI: https://doi.org/10.5753/sbseg_estendido.2025.11864

Abstract

Smart contract security remains a critical challenge on the Ethereum blockchain. This paper investigates the evolution of automated security analysis tools via two experiments leveraging the SmartBugs framework. The first analyzes 215 recently verified Etherscan contracts, focusing on current vulnerability distribution. The second replicates a 2020 study, using its curated dataset of known vulnerabilities but with updated tool versions. Results indicate a lag in the DASP Top 10 taxonomy, and a concerning drop in detection accuracy on the curated dataset—from 41.7% to 24.3%, raising doubts on the tools’ progress.

References

Atzei, N., Bartoletti, M., and Cimoli, T. (2017). A survey of attacks on ethereum smart contracts (sok). Principles of Security and Trust (POST), 10204:164–186.

Bennour, I., Wannes, M., and Ghiss, M. (2024). Enhancing dapp supply chain with verified smart contracts: A case study on the olive-oil industry. In 2024 IEEE/ACS 21st International Conference on Computer Systems and Applications (AICCSA). IEEE.

Casale-Brunet, S., Ribeca, P., Doyle, P., and Mattavelli, M. (2021). Networks of ethereum non-fungible tokens: A graph-based analysis of the erc-721 ecosystem. In 2021 IEEE International Conference on Blockchain (Blockchain), pages 188–195.

Chen, J., Shen, Y., Zhang, J., Li, Z., Grundy, J., Shao, Z., Wang, Y., Wang, J., Chen, T., and Zheng, Z. (2025). Forge: An llm-driven framework for large-scale smart contract vulnerability dataset construction. arXiv preprint arXiv:2506.18795.

Chen, W., Zhang, T., Chen, Z., Zheng, Z., and Lu, Y. (2020). Traveling the token world: A graph analysis of ethereum erc20 token ecosystem. In Proceedings of The Web Conference 2020, WWW ’20, page 1411–1421, New York, NY, USA. Association for Computing Machinery.

Durieux, T., Ferreira, H., Abreu, R., and State, R. (2020a). SmartBugs-curated: Dataset of vulnerable ethereum smart contracts. [link].

Durieux, T., Ferreira, J. F., Abreu, R., and Cruz, P. (2020b). Empirical review of automated analysis tools on 47,587 Ethereum smart contracts. In Proceedings of the ACM/IEEE 42nd International conference on software engineering, pages 530–541.

Eshghie, M., Artho, C., and Gurov, D. (2021). Dynamic vulnerability detection on smart contracts using machine learning.

Etherscan. Verified Contracts - Etherscan. [link].

Feist, J., Grieco, G., and Groce, A. (2019). Slither: A static analysis framework for smart contracts. In 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB), pages 8–15.

Ferreira, J. F., Cruz, P., Durieux, T., and Abreu, R. (2020). SmartBugs: A framework to analyze Solidity smart contracts. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pages 1349–1352.

Grieco, G., Song, W., Cygan, A., Feist, J., and Groce, A. (2020). Echidna: effective, usable, and fast fuzzing for smart contracts. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2020, page 557–560, New York, NY, USA. Association for Computing Machinery.

Grishchenko, I., Maffei, M., and Schneidewind, C. (2018). Foundations and tools for the static analysis of ethereum smart contracts. In Chockler, H. and Weissenbacher, G., editors, Computer Aided Verification, pages 51–78, Cham. Springer International Publishing.

JJ, L. and Singh, K. (2024). Enhancing oyente: four new vulnerability detections for improved smart contract security analysis. International Journal of Information Technology, 16(6):3389–3399.

Kushwaha, S. S., Joshi, S., Singh, D., Kaur, M., and Lee, H.-N. (2022). Ethereum smart contract analysis tools: A systematic review. IEEE Access, 10:57037–57062.

Mehar, M., Shier, C., Giambattista, A., Gong, E., Fletcher, G., Sanayhie, R., Kim, H. M., and Laskowski, M. (2017). Understanding a revolutionary and flawed grand experiment in blockchain: The dao attack. Journal of Cases on Information Technology, 21(1):19–32.

Mossberg, M., Manzano, F., Hennenfent, E., Groce, A., Grieco, G., Feist, J., Brunson, T., and Dinaburg, A. (2019). Manticore: A user-friendly symbolic execution framework for binaries and smart contracts.

Mueller, B. (2018). Smashing ethereum smart contracts for fun and real profit. HITB SECCONF Amsterdam, 9(54):4–17.

NCC Group (2018). Decentralized application security project (dasp) top 10. [link].

OpenZeppelin. Openzeppelin contracts. [link].

Pinna, A., Ibba, S., Baralla, G., Tonelli, R., and Marchesi, M. (2019). A massive analysis of ethereum smart contracts empirical study and code metrics. IEEE Access.

Salzer, G. and Di Angelo, M. (2019). A survey of tools for analyzing ethereum smart contracts.

Staderini, M., Palli, C., and Bondavalli, A. (2020). Classification of ethereum vulnerabilities and their propagations. In 2020 Second International Conference on Blockchain Computing and Applications (BCCA), pages 44–51.

Vidal, F. R., Ivaki, N., and Laranjeiro, N. (2024). Openscv: An open hierarchical taxonomy for smart contract vulnerabilities. Empirical Software Engineering, 29(4):101.

Wang, Y., Lahiri, S. K., Chen, S., Pan, R., and Dillig, I. (2020). Formal verification of workflow policies for smart contracts in azure blockchain. In International Conference on Verified Software: Theories, Tools, and Experiments, pages 230–250. Springer.

Wang, Y., Sheng, S., and Wang, Y. (2023). A Systematic Literature Review on Smart Contract Vulnerability Detection by Symbolic Execution, pages 226–241.

Wood, G. et al. (2014). Ethereum: A secure decentralised generalised transaction ledger. Ethereum project yellow paper, 151(2014):1–32.