michael_nielsen

"The study failed to replicate" is something I often hear, usually with the implication the original study must have been bad, or had something wrong with it. This is a bad misunderstanding of what failure to replicate means.

An example: In 1995, Eric Cornell and Carl Wieman produced an amazing new form of matter, the world's first Bose-Einstein Condensate (BEC) (https://en.wikipedia.org/wiki/Bose%E2%80%93Einstein_condensate … ) They quickly won the Nobel Prize (2001) for this.

In 2000 I was visiting the laboratory of an atomic physicist. His main project was to produce BEC. He'd spent 4 years, perhaps 10,000 hours of staff time (including much of his own), & hundreds of thousands of dollars on this. He'd seen absolutely nothing.

He was somewhat mournful about his failure to see BEC. But the lesson he drew wasn't that the original papers were wrong. It's that he still had more work to do.

I asked him what he thought was wrong. "I don't know. I think it might be a problem with the power supply." He described all the (many, many!) things he was doing to get cleaner current, as well as half a dozen other issues it might be.

Two years later I saw him again, a few months after he'd achieved BEC. I asked what had been the problem. "Turns out it was the power supply!" he said. Not quite beaming -- he's not a beamer -- but as close as he got.

Actually, when I dug down into details, he'd changed or fixed a _lot_ of things in the intervening two years. And it's hard to be sure. Maybe some of those other things were essential, too. Hard to test the counterfactual.

This type of story is very common. Often, failure to replicate means the experimentalist needs to do more work. The source of the trouble is often tacit knowledge or uncontrolled elements in the original paper.