NEJM, arguably the biggest medical journal in the world, has just published this study stating that "remdesivir may have clinical benefit in patients with severe Covid-19" A quick skim says to me that the paper is...bad Let's do a live twitter critical appraisal and see! 1/nhttps://twitter.com/NEJM/status/1248697013870493698 …
-
-
Ok, so, what did the authors actually do? Well, they took a bunch of people who were given remdesivir while very sick with COVID-19 and followed them up for at least 1 day afterpic.twitter.com/cwiIdFKAOd
Show this thread -
Given that the protocol is dated a month after the enrolment of the last patient, we can assume that this was a RETROSPECTIVE trial I.e. all of this treatment was done, then the authors decided to look at the datapic.twitter.com/nWmGWJGyqn
Show this thread -
Nothing wrong with retrospective observational trials, but it means we have to be REALLY cautious about causal conclusions
Show this thread -
The study design was very simple - extremely sick people were given the drug on compassionate grounds (i.e. we don't know if it works but they might die anyway), and then their records were examined to see what happenedpic.twitter.com/ZRbmLhdKiY
Show this thread -
Halfway through the methods, we get this wonderful gem It seems the paper was ghostwritten by a Gilead employee (who is NOT an author) That's an amazing thing to have published in a
@NEJM paper!!!pic.twitter.com/e4NAiDaWnR
Show this thread -
Don't get me wrong, it happens all the time, but rarely is it acknowledged so blatantly in the text It also appears to go against
@NEJM editorial guidelines which...isn't great?pic.twitter.com/yGnjNV9fXr
Show this thread -
Statistical analysis appears reasonable, except for this gem Seems like they're basically saying "because the analysis method we chose couldn't easily accommodate best practice, we didn't do it"pic.twitter.com/oZCk7lD4TS
Show this thread -
Given that this was a RETROSPECTIVE study, it seems likely that they could've just...used a different analysis methodology if they thought that multiple comparisons would be an issue? Weird
Show this thread -
Anyway, on to the results, and this delightful first paragraph Titled "patient randomization" but doesn't talk about randomization (because there wasn't any) The perils of extra-short peer review perhaps?pic.twitter.com/2HffuB5Vw5
Show this thread -
The paragraph also isn't good news for the study's rigor Of 61 total patients, 8 were excluded because of missing information Of the remaining 53, only 40 received the full course of the drug
Show this thread -
Worth pointing out at this point that there's NO CONTROL GROUP That's a huge problem for inference - how do we know if any improvement seen in this trial had anything to do with the drug?
Show this thread -
On to the patient demographics It's a relatively young, although fairly unhealthy population About what you'd expect given the inclusion criteria (although a bit younger perhaps)pic.twitter.com/TKCQ04GwFC
Show this thread -
And here we get to the main results Of the patients (53) treated with remsevidir, most improved! The death rate also appears to have been pretty low at only 13% (7/53)pic.twitter.com/8VogvKfDxU
Show this thread -
Remember, these were VERY sick people. In groups admitted to an ICU for
#COVID19, the median death rate is usually around 30%Show this thread -
The authors have constructed an arbitrary ordinal scale here from 1-6, where 6 is the worst (death) and 1 is the best (discharged) Based on this scale, most patients improved on remdesivirpic.twitter.com/UtcUDlTvzX
Show this thread -
HOWEVER, there was a huge issue here Most patients didn't have the follow-up required to perform this calculation. In fact, based on the IQR presented here, less than 25% had 28 days follow-up data to analyze!pic.twitter.com/asI1gDSlEV
Show this thread -
Younger people did better than older, people who were not mechanically ventilated did better than those who were (not surprising perhaps)pic.twitter.com/2gHltBjWHt
Show this thread -
There were also a large number of reported side-effects, although given the lack of a control group and how sick these people were it's very hard to know if they had anything to do with remdesivirpic.twitter.com/9YS5cmlhxm
Show this thread -
As a fun statistical point, the confidence intervals for some of these regression analyses were, uh, pretty wide Older patients had between 35% and 9417% increase risk of death!pic.twitter.com/HMsGPPYXfs
Show this thread -
So what does this all mean? Well, the authors talk about it in their discussion Apparently, the mortality rate was lower than expected, which is "noteworthy"pic.twitter.com/VCSblvjbs5
Show this thread -
Now, I'd argue that this is...problematic It is extremely difficult to compare patients across trials, and absolutely NOT best practice We also saw a high dropout rate in the trial, with 20% of patients not receiving the complete treatment!https://twitter.com/GidMK/status/1249547200834580485?s=20 …
Show this thread -
We also have very few patients in this trial, and no control group Also, the patients were selected by their doctors - perhaps picking the patients that they thought had a fighting chance? We can't really say whether the death rate was low or high from the data we have!
Show this thread -
To the author's credit, the final paragraph acknowledges most of this!pic.twitter.com/0TdO2drS1C
Show this thread -
Let's sum up:
very small retrospective trial
no control group
written by pharma funder
high dropout
short timeframe
missing data
poorly written/edited
somewhat odd stats
highly selected patient cohort
no causal conclusions!Show this thread -
Basically, it was a very small study with HUGE caveats that showed an interesting possibility Hard to say anything more than that without a proper trial of some kind
Show this thread -
Some might argue that this should not have been published as a research trial, given the many caveats and huge conflicts of interest this study seems to contain I guess that's a question for
@NEJM, who appear to have garnered millions of reads on the article in the last few daysShow this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.