Almost 200 neuroimagers and economists teamed up in this project! An fMRI dataset with 108 participants was distributed to 70 analysis teams. The teams independently analyzed the data with their usual pipeline to test 9 pre-defined hypotheses. How variable were the results? 2/9
-
-
Prikaži ovu nit
-
Fraction of teams reporting a significant binary NHST result varied substantially, with 5 hypotheses falling between 20-40% endorsement across teams. Interestingly, underlying statistical brain maps were more correlated than expected based on the diverse reported results. 3/9
Prikaži ovu nit -
So, even highly correlated statistical maps resulted in different outcomes across teams. We couldn’t test all factors related to variability, but found that spatial smoothness of images, analysis software and correction methods were all significantly related to the results. 4/9
Prikaži ovu nit -
Could researchers in the field predict our results? We used prediction markets and found that researchers in the field are over-optimistic regarding the reproducibility of results, even if they analyzed the data themselves! What’s next? 5/9
Prikaži ovu nit -
Aggregating information across analysis pipelines with meta-analytic approaches is a promising solution and revealed consensus in activated regions across teams for most hypotheses. We’re working on tools to allow this kind of multiverse analysis at the individual study level.6/9
Prikaži ovu nit -
Take home message: sharing uncorrected data will allow for larger-scale tests in the future. One needs to be aware of analytic flexibility in every scientific field with high-dimensional data and multiple complex analytic choices. 7/9
Prikaži ovu nit -
Thanks a lot to all our wonderful collaborators!!! The full list of 197 co-authors is available in the pdf and will soon also appear in the pre-print metadata. 8/9
Prikaži ovu nit -
All analyses in this paper are 100% reproducible (thanks to
@russpoldrack) with open data; see https://github.com/poldrack/narps for details on how to run it yourself. Let us know what you think and add your comment in biorXiv https://www.biorxiv.org/content/10.1101/843193v1 … 9/9Prikaži ovu nit
Kraj razgovora
Novi razgovor -
-
-
You write "The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data." What's your opinion on pre-registration? Some seem to think multipple analyses are not that recommendable.
-
End of page 14 " the use of pre-registration or registered reports can minimize researchers’ degrees of freedom and their effect on neuroimaging results."
- Još 3 druga odgovora
Novi razgovor -
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.