Useful context:https://twitter.com/ronnyk/status/1074409747481223168?s=21 …
Show this thread
New: "p-Hacking & False Discovery in A/B Testing"
Setup: 2,101 RCTs in which businesses track effects over time.
"73% of experimenters stop experiment[s] when a[n] effect reaches 90% confidence...75% of the effects are truly null."
#SocSciResearch
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791 …pic.twitter.com/3QoUCDp3Qp
-
-
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Sensationalized claims. The paper shows that
@Optimizely in 2014 encouraged early stopping ("Our data consists of 2,101 experiments run on the Optimizely A/B testing platform in 2014"). I pointed this out in 2014 review: https://www.amazon.com/gp/customer-reviews/R44BH2HO30T18/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&ASIN=B00E1JO50M … Optimizely addressed it@koomen -
The paper addresses that Optimizely changed procedures in the conclusion sectionpic.twitter.com/eK1d9K1cgu
End of conversation
New conversation -
-
-
This is a sensationalized claim that does not generalize due to heavy selection bias on two axes. I wrote up the details at:https://www.linkedin.com/pulse/p-hacking-ab-testing-sensationalized-ronny-kohavi/ …
- 1 more reply
New conversation -
-
-
@EdMillerPoker Should we not do that? I tell you, I gotta plead ignorance on this thing, because if anyone had said anything to me at all ... that that sort of thing is frowned upon... you know, cause I've worked in a lot of offices, and I tell you, people do that all the time. - 1 more reply
New conversation -
-
-
Thanks for sharing John and Oleg!
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Change the methods and some people will surely find a way to find "significant evidence" when there is not.
-
... more than it is expected by chance.
End of conversation
New conversation -
-
-
@robinson_es think you would be interested in this!Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Bingo...
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.