As an experiment, I'm going to live-tweet the process in this thread throughout the day (w/o identifying any individual papers) to give some insights into common issues raised by reviewers of @RegReports, handling times & so on.
Feel free to mute me if this isn't your thing
/2
-
-
এই থ্রেডটি দেখান
-
1st off the rank an initial Stage 1 submission. Decision: Desk reject w/o in-depth review but invite a resubmission. Main problems: lack of clear link b/w hypotheses, sampling plan & analysis plan, e.g. one power analysis was reported, but many tests proposed (cont') /3
এই থ্রেডটি দেখান -
Crucial that these links are transparent. Also, power analysis fails to meet 90% req in journal policy & a bit wooly which of two DVs (reactions times & error rates) are crucial for confirming/disconfirming hyps, or how conflicting evidence b/w them would be reconciled. /4
এই থ্রেডটি দেখান -
Also, lack of sufficient detail in stated exclusion criteria both within participants & across the sample. These are common problems (see items #2, #3 and #4 below) but o/wise a promising submission. Handling time: 12 calendar days (9 working days). On to the second one... /5pic.twitter.com/NbKBHsGXyy
এই থ্রেডটি দেখান -
Next a revised Stage 1 MS that had received 3 reviews, one +ve & two v critical. Was line-ball after 1st round but where we believe issues are fixable in principle we try to give auths the chance to revise. Auths wrote 10k word R2R. MS returned to revs. Handling time: instant /6
এই থ্রেডটি দেখান -
Third submission. A Stage 2 MS (i.e. with results) reporting a failure to replicate the authors' own previous work. This is a fascinating case for RRs b/c the manipulation check that was necessary to test the specific findings of the original study also failed... (cont') /7
এই থ্রেডটি দেখান -
Now, the only results-based reason for rejecting a Stage 2 RR is -you guessed it- failure of positive controls (including manipulation checks). This provides a tool for rejecting Stage 2 RRs that were conducted poorly but as editors we must wield this v carefully (cont') /8pic.twitter.com/ivTWceNGyF
এই থ্রেডটি দেখান -
Why? B/c balanced against the importance of ensuring methodological quality of published RRs is the need to prevent publication bias in the evidence supporting specific +ve ctrls. A manipulation check could fail b/c the procedure itself is not reliable & that's impt to know. /9
এই থ্রেডটি দেখান -
In such cases, the job of RR editor is to consider (in light of the reviews) the extent to which there is alternative evidence that the study was conducted to a hi standard. In this case I judged that it was, & so should not be rejected b/c of the failed manipulation check. /10
এই থ্রেডটি দেখান -
However, additional discussion of this issue is needed in the paper & possibly some tempering of the conclusions. Outcome: Revise decision. Time under Stage 2 review: 25 calendar days inc. 2 reviewers who assessed it at Stage 1. Lunch and then on to the next one! /11
এই থ্রেডটি দেখান -
4th paper: a journal snafu! It's a Stage 1 RR but the authors submitted it as a regular article which meant the journal administrator assigned the wrong editor, who sent out for review. Took a week before it finally found its way to me. /12
এই থ্রেডটি দেখান -
B/c of this it bypassed the usual RR triage process -and I would have recommended some changes to help increase its odds of success- but since it's in the system already I've assigned extra reviewers to press on. /13
এই থ্রেডটি দেখান -
It received one hi qual rev already, which is remarkable b/c the invitation wd have intro'd it as a regular article. +1 for reviewer dexterity Author tip:when submitting a Registered Report be sure to select the "Registered Report" article type or your MS could go anywhere! /14
এই থ্রেডটি দেখান -
No. 5: Stage 1 MS with 3 positive but critical reviews. The MS includes a precise & comprehensive description of methodology & stats but the big Q from reviewers: does this specific design actually test the broad theory that frames the study? How generalisable is it? (cont') /15
এই থ্রেডটি দেখান -
This sort of Q comes up often w/ Stage 1 RRs. With no results to consider, reviewers often drill deep into theory & the extent to which the study operationalises & measures the concepts proposed by the theory. This presents an interesting & substantial challenge for authors. /16
এই থ্রেডটি দেখান -
Other issues: sometimes w/ mathematically-heavy RRs, authors present info like a tech readout. So some organisational work needed for readabiility. Decision for 5th paper of the day: Major revision, w/ in-depth Stage 1 re-review required. Time in review: 36 calendar days. /17
এই থ্রেডটি দেখান -
Number 6: A new Stage 1 RR, assessed in triage by 4 editors. Falls short in the usual ways: lack of precise hypotheses, lack of reproducible methodological detail, lack of links between hypotheses & analysis plans, insufficient rationale for various methodological elements. /18
এই থ্রেডটি দেখান -
However, the research Q & general approach has promise so we are giving the authors the opportunity to resubmit an improved version. Decision: Desk reject w/ resubmission invited, inc. 1000-word action letter w/ 10-pt feedback from 4 editors. Decision time: 12 calendar days. /19
এই থ্রেডটি দেখান -
Next up: our 7th MS of the day. It's a Stage 1 RR - and in particular a *secondary* RR proposing an analysis of data that already exists but is unseen by the authors. These can be risky endeavours for authors if reviewers critique unchangeable aspects of the study design. /20
এই থ্রেডটি দেখান -
And that's what happened here. The MS was reviewed by 2 experts, w/ one highly critical about the reliability & validity of one of the key measures. This is probably fatal, but in principle that part of the data *might* be recollected or the validity otherwise evidenced... /21
এই থ্রেডটি দেখান -
So we are giving the authors a shot at a Big Ass Revision, just in case they can make a blisteringly strong case for their measure or otherwise salvage it, or replace it. Time under review: 27 calendar days. /22
এই থ্রেডটি দেখান -
And so to No. 8: A new Stage 1 MS. Very tight & clearly written despite proposing complex methods (It's interesting how some 1st-time authors just "get it" when writing RRs while others struggle). We can send this one straight to reviewers. Time at desk: 11 calendar days. /23
এই থ্রেডটি দেখান -
No. 9: a new Stage 1 MS proposing a large multi-site replication. I sent this one out immediately to 3 reviewers who came back w/ positive but v thorough comments covering just about everything. /24
এই থ্রেডটি দেখান -
Some issues that came up: That protocol apparently deviates from the orig study. That analysis plan refers to previous papers but it must be specified to be reproducible. Why the authors use proprietary software. Potential confounds. Whether the resources are worth investing /25
এই থ্রেডটি দেখান -
Decision: Revise. Time under review: 33 days, assessed by 3 reviewers. /26
এই থ্রেডটি দেখান -
And finally, No. 10....oh there is no number 10 (I did it last week, woohoo!) *checks email* MY GOD *hurls laptop into fire* Time to go pick up the kids. Hope y'all found this not too boring. Feel free to weigh in with any Qs or comments. /end (for now)
এই থ্রেডটি দেখান
কথা-বার্তা শেষ
নতুন কথা-বার্তা -
লোড হতে বেশ কিছুক্ষণ সময় নিচ্ছে।
টুইটার তার ক্ষমতার বাইরে চলে গেছে বা কোনো সাময়িক সমস্যার সম্মুখীন হয়েছে আবার চেষ্টা করুন বা আরও তথ্যের জন্য টুইটারের স্থিতি দেখুন।