A systematic review/meta-analysis of honey vs other cough/cold remedies came out yesterday in BMJ Evidence-Based Medicine I think this is worth a quick peer review on twitter 1/npic.twitter.com/XXE2dkbMS9
You can add location information to your Tweets, such as your city or precise location, from the web and via third-party applications. You always have the option to delete your Tweet location history. Learn more
11/n Another issue for the SR/MA is that the data extraction from one of these studies is just wrong Here, honey has a mean reduction of 2.4 in cough frequency compared to 1.2 for 'usual care'pic.twitter.com/ucdT2kwMzO
12/n Now, firstly, that's the reduction not for "honey" but for "honey + coffee", and it's being compared to a control of "coffee alone" which is definitely not what this analysis is looking atpic.twitter.com/tLFg7h58Dn
13/n But further to this, the number given in the meta-analysis of -2.4 is wrong, as it should be 3.0-0.4 = -2.6 So it's wrong in two ways, both of them very bad for this analysis
14/n I should also mention, both of these studies were rated as at low/moderate risk of bias in the systematic review. Have a look yourself and see if you think that this is reasonable: https://www.sid.ir/en/journal/ViewPaper.aspx?ID=196449 … https://www.nature.com/articles/pcrj201372#Sec9 …pic.twitter.com/UnJ3TNif3y
15/n I haven't gone through every study yet, but every one that I have is similarly worrying. I can't actually find a study that I'm not worried about in this analysis
16/n If I go through the studies, I really can't find any that I would rate as a "low" risk of bias. Most of them are "high", some of them are really worryingly bad
17/n Given this, I'm not sure the conclusions of the review make much sense at all. How can you interpret studies of this quality as moderate/strong evidence???pic.twitter.com/mvlxiUAvJX
18/n I'm also worried about the heterogeneity of the 'usual care' groups across studies. Some of the usual care: - coffee - paracetamol - syrup - diphenhydramine - placebo - prednisolone These are very much not the same!
19/n That's a big issue because you can't really combine the effect of coffee with diphenhydramine and expect it to make sense, but that's what the authors did
20/n Ultimately, I think the only real conclusion you can draw here is that we have very little idea whether honey reduces symptoms for URTI/cough, and that the research is quite problematic
21/n Not something that'll make headlines, perhaps, but sadly that's often how these things go
22/n N.B. this study has already hit ~800 on Altmetric, been covered internationally, and made huge newspic.twitter.com/bj72yqShfA
23/n Something else I didn't mention. Every study that I've looked at so far used a per-protocol analysis, which is a huge and worrying issue All of these should be at a high risk of attrition bias, yet none were rated as such. Most of them were green (low risk)pic.twitter.com/Y6sXlnkJHe
24/n I've gotta say, for anyone teaching students about how finicky bias can be in systematic reviews, this is a beautiful example of getting it wrong
25/n Somehow there are more issues here. This study was included in the risk of bias, but even though it assessed honey vs placebo/salbutamonl (and found no effect) it is not in any of the meta-analyses Very weirdpic.twitter.com/8AaMA85ERm
26/n This is even weirder when you consider that the authors report excluding a study for not providing data So they exclude one study and report it, but another just...disappears? So strangepic.twitter.com/oyrp57yP1Y
27/n Another included study using a per-protocol analysis. This was at least rated correctly as at a high risk of biaspic.twitter.com/t2mRMIq539
28/n Another one. This study was rated at low risk of bias for most domains. Here's how they described their randomization and allocation concealment. What do you think?pic.twitter.com/1NM606xcRg
29/n I'll give you a head-start - if they literally don't report ~how~ patients were randomized, by definition this should be unclear or high risk of bias for the domain of random sequence generation
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.