Smart Contracts Bug Finding Survey


id: "dummy" password: "crowd123"

SCRUBD Dataset

Smart Contracts Reentrancy and Unhandled Exceptions Vulnerability Dataset

Overview

SCRUBD is a comprehensive dataset of Ethereum smart contracts labeled for two critical vulnerabilities:

  • Reentrancy (RE)
  • Unhandled Exceptions (UX)

SCRUBD/CD

Crowdsourced Dataset

  • Total RE Functions: 746, Vulnerable: 245, Non-Vulnerable: 501
  • Total UX Functions: 566, Vulnerable: 275, Non-Vulnerable: 291

SCRUBD/SD

Synthesized Dataset

  • 239 crafted test cases
  • Vulnerable: 155, Non-Vulnerable: 84
  • Covers corner cases

Key Features

  • ✅ Real-world contracts from Ethereum mainnet (via Google BigQuery)
  • ✅ Rigorous labeling through crowdsourcing + expert validation
  • ✅ Synthetic test cases covering RE edge cases
  • ✅ Tool evaluation comparing 6 analysis tools (Slither, Mythril, etc.)
  • ✅ Detailed documentation with vulnerability patterns

Access the Dataset

GitHub Repository | arXiv Paper | Crowdsourcing Platform

Research Findings

Tool Performance Comparison (F1-Scores):

Tool Performance Comparison

Figure: F1-Scores of vulnerability detection tools on SCRUBD
Slither (SL), Sailfish (SF), Solhint (SH), Mythril (MY), Conkas (CK), SmartCheck (SC)

Key Results:

  • Slither performs best on real-world contracts (SCRUBD/cD) for both RE (0.61) and UX (0.83)
  • Sailfish excels on synthetic edge cases (SCRUBD/sD) for RE (0.75)
  • Sailfish doesn't support UX detection (N/A)

Applications

  • 🔬 Tool Evaluation: Benchmark RE/UX detection tools
  • 🤖 ML Training: Train vulnerability prediction models
  • 📊 Empirical Studies: Analyze vulnerability patterns

How to Cite

Chavhan Sujeet Yashavant, MitrajSinh Chavda, Saurabh Kumar, Amey Karkare, and Angshuman Karmakar. "SCRUBD: Smart Contracts Reentrancy and Unhandled Exceptions Vulnerability Dataset" in 22nd International Conference on Mining Software Repositories (Data and Tool Showcase Track), 2025. [arXiv]