A systematic review/meta-analysis of honey vs other cough/cold remedies came out yesterday in BMJ Evidence-Based Medicine I think this is worth a quick peer review on twitter 1/npic.twitter.com/XXE2dkbMS9
You can add location information to your Tweets, such as your city or precise location, from the web and via third-party applications. You always have the option to delete your Tweet location history. Learn more
8/n But it doesn't take long to find issues. There are large tracts of text that are identical between the studies. There are numerical errors in the tables and GRIM inconsistencies ping @sTeamTraen @MicrobiomDigest (rounding done wrong and incorrect totals).pic.twitter.com/oUZMb3RXqB
9/n Moreover, the risk of bias assessment in the systematic review is, as far as I can tell, incorrect. The authors of these studies do not report, for example, how the randomization procedure was generatedpic.twitter.com/B4C8gf8D3d
10/n They also don't describe blinding of clinical staff, and actually from reading I'm fairly sure the doctors could tell which medication the patients were taking
11/n Another issue for the SR/MA is that the data extraction from one of these studies is just wrong Here, honey has a mean reduction of 2.4 in cough frequency compared to 1.2 for 'usual care'pic.twitter.com/ucdT2kwMzO
12/n Now, firstly, that's the reduction not for "honey" but for "honey + coffee", and it's being compared to a control of "coffee alone" which is definitely not what this analysis is looking atpic.twitter.com/tLFg7h58Dn
13/n But further to this, the number given in the meta-analysis of -2.4 is wrong, as it should be 3.0-0.4 = -2.6 So it's wrong in two ways, both of them very bad for this analysis
14/n I should also mention, both of these studies were rated as at low/moderate risk of bias in the systematic review. Have a look yourself and see if you think that this is reasonable: https://www.sid.ir/en/journal/ViewPaper.aspx?ID=196449 … https://www.nature.com/articles/pcrj201372#Sec9 …pic.twitter.com/UnJ3TNif3y
15/n I haven't gone through every study yet, but every one that I have is similarly worrying. I can't actually find a study that I'm not worried about in this analysis
16/n If I go through the studies, I really can't find any that I would rate as a "low" risk of bias. Most of them are "high", some of them are really worryingly bad
17/n Given this, I'm not sure the conclusions of the review make much sense at all. How can you interpret studies of this quality as moderate/strong evidence???pic.twitter.com/mvlxiUAvJX
18/n I'm also worried about the heterogeneity of the 'usual care' groups across studies. Some of the usual care: - coffee - paracetamol - syrup - diphenhydramine - placebo - prednisolone These are very much not the same!
19/n That's a big issue because you can't really combine the effect of coffee with diphenhydramine and expect it to make sense, but that's what the authors did
20/n Ultimately, I think the only real conclusion you can draw here is that we have very little idea whether honey reduces symptoms for URTI/cough, and that the research is quite problematic
21/n Not something that'll make headlines, perhaps, but sadly that's often how these things go
22/n N.B. this study has already hit ~800 on Altmetric, been covered internationally, and made huge newspic.twitter.com/bj72yqShfA
23/n Something else I didn't mention. Every study that I've looked at so far used a per-protocol analysis, which is a huge and worrying issue All of these should be at a high risk of attrition bias, yet none were rated as such. Most of them were green (low risk)pic.twitter.com/Y6sXlnkJHe
24/n I've gotta say, for anyone teaching students about how finicky bias can be in systematic reviews, this is a beautiful example of getting it wrong
25/n Somehow there are more issues here. This study was included in the risk of bias, but even though it assessed honey vs placebo/salbutamonl (and found no effect) it is not in any of the meta-analyses Very weirdpic.twitter.com/8AaMA85ERm
26/n This is even weirder when you consider that the authors report excluding a study for not providing data So they exclude one study and report it, but another just...disappears? So strangepic.twitter.com/oyrp57yP1Y
27/n Another included study using a per-protocol analysis. This was at least rated correctly as at a high risk of biaspic.twitter.com/t2mRMIq539
28/n Another one. This study was rated at low risk of bias for most domains. Here's how they described their randomization and allocation concealment. What do you think?pic.twitter.com/1NM606xcRg
29/n I'll give you a head-start - if they literally don't report ~how~ patients were randomized, by definition this should be unclear or high risk of bias for the domain of random sequence generation
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.