Anne Scheel

@annemscheel

PhD student trying to figure out how we can make psych research more reproducible at . Co-mod of , co-blogger at

Eindhoven, Nederland
Vrijeme pridruživanja: lipanj 2011.

Tweetovi

Blokirali ste korisnika/cu @annemscheel

Jeste li sigurni da želite vidjeti te tweetove? Time nećete deblokirati korisnika/cu @annemscheel

  1. Prikvačeni tweet
    5. velj

    New preprint! Mitchell Schijen, , and I compared the rate of "positive" results (i.e., confirmed hypotheses) in Registered Reports to a sample of standard (non-RR) papers in psychology. We found a *very* large difference. Thread... 1/

    Prikaži ovu nit
    Poništi
  2. proslijedio/la je Tweet
    prije 10 sati

    This paper on Registered Reports is excellent, and it is hard to express how delighted I am to be able to update my MSc course materials on reproducibility to include work by a former student!

    Poništi
  3. proslijedio/la je Tweet

    on and 's new study finding that Registered Reports are about 50% likely to confirm their hypotheses, and why that faintly boring technical-sounding result is super important and starkly reveals the problems with science

    Poništi
  4. proslijedio/la je Tweet
    prije 12 sati
    Poništi
  5. proslijedio/la je Tweet
    3. velj

    You know that person who writes great reviews/comments? Who you’re always wondering what they think of the latest paper you read (or wrote)? Encourage them to apply to be an Associate Editor at ! All ranks & areas of psych are welcome to apply! We want a diverse team!

    Poništi
  6. proslijedio/la je Tweet
    prije 23 sata

    In these increasingly divided times when factions that generally agree with each other are turning against each other I just want to remind people that it’s possible to see the good in both AND

    Poništi
  7. 5. velj

    3) I can't believe how long it took to write this up -- (transparent) science is hard. Thanks to everyone who helped with feedback and inspiration! Especially 15/15

    Prikaži ovu nit
    Poništi
  8. 5. velj

    2) Study is preregistered & we share all data/code/materials in a way that hopefully allows full reproducibility & replicability. I'd be thrilled if ppl used our data to look into other fun stuff - it has quotes of all coded hypotheses & conclusions! 14/

    Prikaži ovu nit
    Poništi
  9. 5. velj

    1) You may have seen a previous version of our results floating around Twitter, from a conference presentation last year. The numbers changed slightly since then and should be considered outdated (small errors, replaced exclusions, revised a variable) 13/

    Prikaži ovu nit
    Poništi
  10. 5. velj

    I have a whole other story to tell about an exploratory qualitative analysis we ran on the language used to describe hypothesis in RRs, but I'm tired and this'll have to wait (or why don't you go read the paper and see for yourself? 😉). Final words: 12/

    Prikaži ovu nit
    Poništi
  11. 5. velj

    And this is important to keep in mind: We did not evaluate if RRs "work"! This is an observational study that probably contains many confounds. To get to the bottom of RR efficacy, we'd need a randomised controlled trial, or something close(r) to it. 11/

    Prikaži ovu nit
    Poništi
  12. 5. velj

    For me personally, 43-50% positive results still sounds like too much. It could be due to the method that gives authors some wiggle room to claim support for hypotheses with disappointing results. Still, it may also mean that the RR format can't completely eradicate all bias 10/

    Prikaži ovu nit
    Poništi
  13. 5. velj

    But what we *can* say: The positive result rate in the standard literature is incompatible with any reasonable prior probability hypotheses might have. This plot shows that standard papers would need to test >90% true hypotheses with >90% power if we don't assume bias! 9/

    Prikaži ovu nit
    Poništi
  14. 5. velj

    Still, even when we're looking only at novel studies, RR hypotheses might really have a lower prior: In-principle acceptance means that authors don't need to worry about getting unpublishable null results, so maybe they use RRs for riskier hypotheses. We can't rule that out. 8/

    Prikaži ovu nit
    Poništi
  15. 5. velj

    You might say: Maybe RRs are testing hypotheses with a lower prior probability? Aren't tons of RRs just failed replications? Yes, more than half of RRs were replications, and yes, those had even fewer positive results. But the effect is still huge when we exclude them: 7/

    Prikaži ovu nit
    Poništi
  16. 5. velj

    Our sample from the standard literature is a direct replication of Fanelli (2010), except limited to the publication years 2013-2018. We find 96.05% positive results, confirming the sky-high success rate. Compare that to only 43.66% in RRs! 6/

    Prikaži ovu nit
    Poništi
  17. 5. velj

    It's quick & dirty: Identify the first hypothesis (for RRs: first *preregistered* hypothesis) mentioned in the abstract or full text of each paper and code if the authors concluded that it was supported (fully or partially) or not. 5/

    Prikaži ovu nit
    Poništi
  18. 5. velj

    So if RRs are successful at reducing pub bias and QRPs, we should expect them to have a (much) lower positive result rate than the standard literature! To test that, we used the same method as Fanelli (2010, ), who found 91.5% positive results in psych. 4/

    Prikaži ovu nit
    Poništi
  19. 5. velj

    We've known that psychology has impossibly high "success rates" since the 1950s -- the literature is extremely positive, while stat. power is very mediocre. The likely explanation is that null results are either not published (pub bias) or hacked into positive results (QRPs) 3/

    Prikaži ovu nit
    Poništi
  20. 5. velj

    were designed to minimise publication bias & QRPs -- so I've been wanting to see how they do on some indicators of pub bias for ages. (As finds in his cool new paper, even "annoying" side effects of RRs might be beneficial! ) 2/

    Prikaži ovu nit
    Poništi

Čini se da učitavanje traje već neko vrijeme.

Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.

    Možda bi vam se svidjelo i ovo:

    ·