michael_nielsen

"The study failed to replicate" is something I often hear, usually with the implication the original study must have been bad, or had something wrong with it. This is a bad misunderstanding of what failure to replicate means.

An example: In 1995, Eric Cornell and Carl Wieman produced an amazing new form of matter, the world's first Bose-Einstein Condensate (BEC) (https://en.wikipedia.org/wiki/Bose%E2%80%93Einstein_condensate … ) They quickly won the Nobel Prize (2001) for this.

In 2000 I was visiting the laboratory of an atomic physicist. His main project was to produce BEC. He'd spent 4 years, perhaps 10,000 hours of staff time (including much of his own), & hundreds of thousands of dollars on this. He'd seen absolutely nothing.

He was somewhat mournful about his failure to see BEC. But the lesson he drew wasn't that the original papers were wrong. It's that he still had more work to do.

I think the crucial difference here is that by 2000 dozens of other labs besides Cornell and Wieman had ALSO made BEC. A better physics analogue to modern crises would be a genuinely ambiguous case like the "supersolid" thing.

In the counterfactual world where that hadn't happened, it wouldn't change the point: failure to immediately replicate != bad. Consider if the analogous thing happened with the Higgs.

I think this is a case where it's risky to generalize too much from physics. The tightly controlled nature of physics experiments and the high statistical certainty obtained thereby makes "problem with the original" much less likely than for a p=0.049 psych study.