The paper clearly acknowledges our move to sequential testing in 2015. The results are notable for any organization using a fixed-horizon approach w software that does not strictly enforce a pre-determined stopping time. Many commercial and proprietary tools still don't.
Claim of widespread p-hacking sensationalized in paper suffering lack of external validity on two axes: skew due to massive selection bias and time bias.
My response in https://www.linkedin.com/pulse/p-hacking-ab-testing-sensationalized-ronny-kohavi/ …
@Optimizely @dsiroker @koomen @marketsensei @LPekelishttps://twitter.com/JohnHolbein1/status/1074091992836009984 …
-
-
- 1 more reply
New conversation -
-
-
In addition to telling users to stop based on pvalue, the software vendor presented (1-pvalue) as the posterior probability of the alternative. It is not unreasonable behavior by clients to stop a test when believing alternative has at least 90% chance to beat control.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.