Before I begin, I think the MAIAN analysis tool is a useful advance to the field and we plan to incorporate some of their features into Manticore in the coming months. My main issue is with their evaluation of results.
-
-
Show this thread
-
In their paper, a “greedy contract” is any contract that can lock ethers. However, ALL CONTRACTS compiled with solc <= 0.3.6 without a ‘withdraw’ function are “greedy contracts” and are included in their totals.
Show this thread -
This is because solc < 0.4.0 allowed any function to receive ethers. This behavior has since changed and current versions of solc use the “payable” attribute to denote functions that can receive ether.
Show this thread -
To really drive this home, even their example “true positive” issue in section 5.2 is a greedy contract due to this behavior of solc. This vastly inflates their numbers beyond what reasonably should get counted.
Show this thread -
Next issue: the authors indicate they found only 1,524 _distinct_ greedy contracts. ~30k comes from instances of the issues. There is no discussion of the distribution. Are there a few contracts that repeat the vast majority of these issues? We don’t know!
Show this thread -
Even further skewing their results, they neglect to discuss that _many_ contracts deployed to public chains are there only for testing. Better benchmarks would account for usage.
Show this thread -
In our own testing at
@trailofbits, we have tools (e.g., Slither and others) that have detected _thousands_ of vulnerabilities on the public Ethereum blockchain, but most of the contracts affected are not used or are useless.Show this thread -
We think our work contributes to the field and have found many issues for our clients with them, but we have resisted fantastical headlines like the ones in this paper because we are aware of the caveats.
Show this thread -
Many of the articles written about this paper have neglected to note that it was never accepted to a peer-reviewed journal, a process which exists to root out exactly these issues of bias and methodology.
Show this thread -
Further, many journalists writing about this paper have only interviewed its authors for commentary, neglecting to speak with industry experts. Please consider myself and the team
@trailofbits a source when writing about blockchain security!Show this thread -
In order to move this conversation forward, we’re calling on the MAIAN team to release the raw data they used to discover these issues, or share it privately with a qualified team capable of reviewing it.
Show this thread -
Oops! Lost in Twitter's threaded replies: "For example, they could have only counted contracts with at least X transactions or contracts that received more than Y ethers. Defining some kind of low watermark is essential for reviewing contracts on public chains."
Show this thread -
Quoting a few tweets so they show up in threaded replies: "The only contract address cited in this paper never had any ether sent to it"https://twitter.com/computerality/status/966802544247869440 …
Show this thread -
Ethereum mainnet was once used as a testnethttps://twitter.com/maurelian_/status/966810648171765760 …
Show this thread -
It's 2018. If you publish research, publish your data too.https://twitter.com/defendtheworld/status/966801561694490628 …
Show this thread
End of conversation
New conversation -
-
-
Read the paper yesterday as part of my research for my master thesis, indeed I found the interpretation of the results to be skewed. Thank you for the insight.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.