"The study failed to replicate" is something I often hear, usually with the implication the original study must have been bad, or had something wrong with it. This is a bad misunderstanding of what failure to replicate means.
Show this thread
-
Actually, when I dug down into details, he'd changed or fixed a _lot_ of things in the intervening two years. And it's hard to be sure. Maybe some of those other things were essential, too. Hard to test the counterfactual.
Show this thread
This type of story is very common. Often, failure to replicate means the experimentalist needs to do more work. The source of the trouble is often tacit knowledge or uncontrolled elements in the original paper.
-
-
This most emphatically doesn't mean the original paper is bad. Indeed, as with BEC, the original paper may be extremely good. Instead, it may mean more work is needed to understand exactly what's required to see the effect. The original paper is merely an important first step.
Show this thread -
Another good example: measurements of the quality factor of sapphire (basically, how good a lasing substance is it) differed by _orders of magnitude(!)_ between Russia and the West during the cold war.
Show this thread -
It took more than 20 years to sort this out! Turns out it was due to tacit knowledge available in the Russian lab that wasn't known in the West. Story is told here: https://orca.cf.ac.uk/71069/1/wrkgpaper1.pdf …pic.twitter.com/qFJFRRG6er
Show this thread -
It's worth noting: some Western scientists thought this meant the Russian results were wrong. Turns out it was the Westerners who were wrong. (The Q of sapphire was a hot topic, as it was thought to be relevant for the detection of gravitational waves. So, not small stakes.)pic.twitter.com/UVP36d9eVw
Show this thread -
A tempting response is to say "Oh, the paper should have included more detail." But first-rate experiments often include a mindboggling number of details that have to be gotten right. Figuring those out is (rightly) the decades-long task of an entire community doing followup work
Show this thread -
If you don't believe this, look at the miniscule details Collins paper on the Q of sapphire. Or write out a list of all the possible noise sources in your power supply that might muck up an experiment. (I'll be waiting when your list passes 100 items.)
Show this thread -
The "failure to replicate = bad" narrative is tempting. But it's a dramatic misunderstanding & oversimplification of how science works. I wish people had better mental models, to understand that failure to replicate is often instead merely a step along the way to understanding.
Show this thread
End of conversation
New conversation -
-
-
My own experience (and I'm not a grown-up scientist) is that for real-world experiments, as allegedly for small boys, cleanliness is next to impossible.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.