NEJM, arguably the biggest medical journal in the world, has just published this study stating that "remdesivir may have clinical benefit in patients with severe Covid-19" A quick skim says to me that the paper is...bad Let's do a live twitter critical appraisal and see! 1/nhttps://twitter.com/NEJM/status/1248697013870493698 …
-
-
Don't get me wrong, it happens all the time, but rarely is it acknowledged so blatantly in the text It also appears to go against
@NEJM editorial guidelines which...isn't great?pic.twitter.com/yGnjNV9fXr
Show this thread -
Statistical analysis appears reasonable, except for this gem Seems like they're basically saying "because the analysis method we chose couldn't easily accommodate best practice, we didn't do it"pic.twitter.com/oZCk7lD4TS
Show this thread -
Given that this was a RETROSPECTIVE study, it seems likely that they could've just...used a different analysis methodology if they thought that multiple comparisons would be an issue? Weird
Show this thread -
Anyway, on to the results, and this delightful first paragraph Titled "patient randomization" but doesn't talk about randomization (because there wasn't any) The perils of extra-short peer review perhaps?pic.twitter.com/2HffuB5Vw5
Show this thread -
The paragraph also isn't good news for the study's rigor Of 61 total patients, 8 were excluded because of missing information Of the remaining 53, only 40 received the full course of the drug
Show this thread -
Worth pointing out at this point that there's NO CONTROL GROUP That's a huge problem for inference - how do we know if any improvement seen in this trial had anything to do with the drug?
Show this thread -
On to the patient demographics It's a relatively young, although fairly unhealthy population About what you'd expect given the inclusion criteria (although a bit younger perhaps)pic.twitter.com/TKCQ04GwFC
Show this thread -
And here we get to the main results Of the patients (53) treated with remsevidir, most improved! The death rate also appears to have been pretty low at only 13% (7/53)pic.twitter.com/8VogvKfDxU
Show this thread -
Remember, these were VERY sick people. In groups admitted to an ICU for
#COVID19, the median death rate is usually around 30%Show this thread -
The authors have constructed an arbitrary ordinal scale here from 1-6, where 6 is the worst (death) and 1 is the best (discharged) Based on this scale, most patients improved on remdesivirpic.twitter.com/UtcUDlTvzX
Show this thread -
HOWEVER, there was a huge issue here Most patients didn't have the follow-up required to perform this calculation. In fact, based on the IQR presented here, less than 25% had 28 days follow-up data to analyze!pic.twitter.com/asI1gDSlEV
Show this thread -
Younger people did better than older, people who were not mechanically ventilated did better than those who were (not surprising perhaps)pic.twitter.com/2gHltBjWHt
Show this thread -
There were also a large number of reported side-effects, although given the lack of a control group and how sick these people were it's very hard to know if they had anything to do with remdesivirpic.twitter.com/9YS5cmlhxm
Show this thread -
As a fun statistical point, the confidence intervals for some of these regression analyses were, uh, pretty wide Older patients had between 35% and 9417% increase risk of death!pic.twitter.com/HMsGPPYXfs
Show this thread -
So what does this all mean? Well, the authors talk about it in their discussion Apparently, the mortality rate was lower than expected, which is "noteworthy"pic.twitter.com/VCSblvjbs5
Show this thread -
Now, I'd argue that this is...problematic It is extremely difficult to compare patients across trials, and absolutely NOT best practice We also saw a high dropout rate in the trial, with 20% of patients not receiving the complete treatment!https://twitter.com/GidMK/status/1249547200834580485?s=20 …
Show this thread -
We also have very few patients in this trial, and no control group Also, the patients were selected by their doctors - perhaps picking the patients that they thought had a fighting chance? We can't really say whether the death rate was low or high from the data we have!
Show this thread -
To the author's credit, the final paragraph acknowledges most of this!pic.twitter.com/0TdO2drS1C
Show this thread -
Let's sum up:
very small retrospective trial
no control group
written by pharma funder
high dropout
short timeframe
missing data
poorly written/edited
somewhat odd stats
highly selected patient cohort
no causal conclusions!Show this thread -
Basically, it was a very small study with HUGE caveats that showed an interesting possibility Hard to say anything more than that without a proper trial of some kind
Show this thread -
Some might argue that this should not have been published as a research trial, given the many caveats and huge conflicts of interest this study seems to contain I guess that's a question for
@NEJM, who appear to have garnered millions of reads on the article in the last few daysShow this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.