@peterseibel I tried to update this to use http://www.evanmiller.org/bayesian-ab-testing.html … as the stopping condition, but empirically still get the peeking problem.
-
-
@avibryant Hmmmm. I guess one of us three isn't as clever as he thinks he is. -
@peterseibel@avibryant Using flat priors won't change things much. -
@peterseibel@avibryant actually kinda jumped the gun there. Probably nothing can help when there're egregious stopping rules. -
@venusatuluri@peterseibel yeah, which is fine. But you frequently see assertions that "bayesian exp design doesn't have that problem". -
@venusatuluri@peterseibel so I'm trying to understand what that actually means, and how I can empirically demonstrate it. -
@avibryant@peterseibel Rather the point is to estimate the effect size and the uncertainty there, instead of giving a binary decision. -
@venusatuluri@avibryant So how do you run an experiment so you can stop as soon as you reach a certain acceptable level of uncertainty. -
@peterseibel@venusatuluri presumably by modeling the EV of doing so, and doing it only when it seems worthwhile. - 2 more replies
New conversation -
-
-
@peterseibel maybe I'm thinking about stopping wrong; naively, just going for "probability B beats A is > 97.5% or < 2.5%". cc@EvMill -
@peterseibel@EvMill here's my hacky diff of the interruptus code: https://gist.github.com/avibryant/bf4456bb84a2206cf783 … -
@avibryant@peterseibel@EvMill damn timelines I think I'm cleverly connecting the discussions, really I'm hours behind.
End of conversation
New conversation -
-
-
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.