Ken Kocienda

When I think of the creative work that means the most to me: the music of Bach and Pink Floyd, the writings of Shakespeare and Lao-Tze, the art of Diane Arbus and Jasper Johns… Were any of these created with A/B testing? Nope.https://twitter.com/kocienda/status/1113450945705177088 …

I'm proud of the work I did with my colleagues at Apple, but I don't claim to be right about everything, or anything, when it comes to the best method for doing great creative and technical work.

My biggest bug bear with a/b testing is how you resolve the issue of whether or not a short-term positive will actually turn into a long term negative. Impossible to isolate over the longer term so you’re acting on short-term signals that may cause longer term damage.

Yep. Placing blind trust in the results of A/B tests means ignoring the bigger picture.

I don’t disagree that there are places where A/B testing is used instead of judgement, but it’s not a practice you can completely discount.

Is an A/B test ever the right choice? Well, what are you optimizing for? At Apple, we wanted the best experience for people, never the most clicks, and since the dynamic range of the choices is often small, we just made a call. We didn't test things that didn't matter to us.

How much of this has to do with the nature of the product you worked on? Are you saying that Apple doesn’t use A/B testing on other products, such as Apple Store and others than are more transaction-oriented?

I can only speak for my own experience on Mac and iOS software. I honestly don’t know what other parts of the company did.