Sybil Report on Gitcoin Grants Rounds 11

tags: gitcoin Reports

Updated by September 2021

Authors: Danilo Lessa Bernardineli

Summary

The combined FDD process is effective at catching about 83% of the Sybil Users (between 100% to 57% under 95% CI) according to blind human evaluations. The best estimate for Sybil Incidence on Gitcoin is 6.4%, and is contained between 3.6% to 9.3% with 95% CI. IP clusters together with GitHub account creation date are the most relevant features for detecting sybil users programatically right now. The Fraud Tax is computed as 0.61% of the Funding Amount.

Links

• Sybil Detection Service Repo: https://github.com/gitcoindao/gitcoin_asop/tree/2c39f443068a9378ab9aadb5b7364e53a178cdbb
  • Summary stats: https://github.com/gitcoindao/gitcoin_asop/blob/2c39f443068a9378ab9aadb5b7364e53a178cdbb/notebooks/gr11_report.ipynb
  • Incidence stats: https://github.com/gitcoindao/gitcoin_asop/blob/2c39f443068a9378ab9aadb5b7364e53a178cdbb/notebooks/gr11_sybil_incidence.ipynb
  • List of flagged users: https://github.com/gitcoindao/gitcoin_asop/blob/2c39f443068a9378ab9aadb5b7364e53a178cdbb/data/predict_data/2021-09-27T18:59:10%2B00:00/2021-09-27T19:03:53-gc_df.csv

Parameters

Algorithm Aggressiveness: 30%
Matching Pool: 950k USD

Algorithm Aggressiveness tunes how much sensitive / specific the flagging model should be. When it is set to 50%, then it will be optimized for accuracy, while closer values to 0% means that it will be optimized for minimizing false detection rate

Trained Model Feature Importance

Statistics

Sybil Incidence & Detection

Sybil Incidence

• Estimated Unbiased Sybil Incidence: 6.4% (between 3.6% and 9.3%, 95% CI)
• Esimated Sybil Users (unbiased): 1127 (between 635 and 1631, 95% CI)
• Flagged Users Fraction: 5.3%
• Flagged Sybil Users: 853
• Can we reject the hypothesis that we're catching everyone? No
• Best estimate for how much % of the sybil users are being caught: 82.9%
• Worst estimate for how much % of the sybil users are being caught: 56.9%

The Unbiased Sybil Incidence is the estimated incidence by debiasing the probability parameter as estimated by a Binomial Distribution over the Average is_sybil on the Human Evaluation flags. Debiasing takes into account the squelches and the sampling distribution.

Sybil Flags

• ML generated flags: 494
• Heuristic generated flags: 178
• Human provided flags: 1247
• Total flags: 1247

Flags can be duplicate for the same user if they're evaluated more than once

Summary Statistics on Scenarios

Contributors

Total Contributors (original): 15986
Total Contributors (modified): 15134
Total Contributors (removed): 853

Sum of USDT Amount (original): 3499639.63
Sum of USDT Amount (removed): 606494.88

Median of Median USDT Amount per User (original): 1.43
Median of Median USDT Amount per User (modified): 1.44
Median of Median USDT Amount per User (removed): 1.36

Median Contribution Count per User (original): 17.00
Median Contribution Count per User (modified): 17.00
Median Contribution Count per User (removed): 20.00

Mean Count per User (original): 27.89
Mean Count per User (modified): 29.46
Mean Count per User (removed): 34.45

Quadratic Funding Statistics

Total Grant Funding (original): 950000.00
Total Grant Funding (modified): 950000.00

Mean Grant Funding (USDT, original): 769.85
Mean Grant Funding (USDT, modified): 780.61

Median Grant Funding (USDT, original): 0.02
Median Grant Funding (USDT, modified): 0.02

Grant count (original): 1234
Grant count (modified): 1217

Max payout (USDT) 955787.22
Fraud Tax (USDT) 5787.22
Fraud Tax (% of total funding) 0.61%