2/n The review is here:https://ebm.bmj.com/content/early/2020/07/28/bmjebm-2020-111336 …
-
-
Show this thread
-
3/n Basically, the authors searched the literature for all the studies looking at honey being used to treat URTIs (cough), and aggregated them together into one model. They found that honey works better than usual care! Amazing right?pic.twitter.com/BvZBO8u9Yu
Show this thread -
3.5/n (N.B. URTI = Upper Respiratory Tract Infection)
Show this thread -
4/n And if you read the methods section of this review, it's fairly reasonable. Good exclusion criteria, risk of bias assessment, it ticks all the boxes! But as ever, the devil is in the detailspic.twitter.com/qFUyVGbAFV
Show this thread -
5/n At first glance, the results look rosy. Most studies were assessed as of reasonable quality, and the overall benefits seem accurate if very weak due to the low overall sample sizepic.twitter.com/ELAhFavao2
Show this thread -
6/n The thing about systematic reviews/meta-analyses is that the included studies are EVERYTHING As they say, garbage in = garbage out Let's take 2 of these studies as an examplepic.twitter.com/aAROQ7bphu
Show this thread -
7/n These studies are both by the same author group, published close together, and both report finding that honey works better than the alternative to treat cough over 1 weekpic.twitter.com/Lyv7cqUUjI
Show this thread -
8/n But it doesn't take long to find issues. There are large tracts of text that are identical between the studies. There are numerical errors in the tables and GRIM inconsistencies ping
@sTeamTraen@MicrobiomDigest (rounding done wrong and incorrect totals).pic.twitter.com/oUZMb3RXqB
Show this thread -
9/n Moreover, the risk of bias assessment in the systematic review is, as far as I can tell, incorrect. The authors of these studies do not report, for example, how the randomization procedure was generatedpic.twitter.com/B4C8gf8D3d
Show this thread -
10/n They also don't describe blinding of clinical staff, and actually from reading I'm fairly sure the doctors could tell which medication the patients were taking
Show this thread -
11/n Another issue for the SR/MA is that the data extraction from one of these studies is just wrong Here, honey has a mean reduction of 2.4 in cough frequency compared to 1.2 for 'usual care'pic.twitter.com/ucdT2kwMzO
Show this thread -
12/n Now, firstly, that's the reduction not for "honey" but for "honey + coffee", and it's being compared to a control of "coffee alone" which is definitely not what this analysis is looking atpic.twitter.com/tLFg7h58Dn
Show this thread -
13/n But further to this, the number given in the meta-analysis of -2.4 is wrong, as it should be 3.0-0.4 = -2.6 So it's wrong in two ways, both of them very bad for this analysis
Show this thread -
14/n I should also mention, both of these studies were rated as at low/moderate risk of bias in the systematic review. Have a look yourself and see if you think that this is reasonable: https://www.sid.ir/en/journal/ViewPaper.aspx?ID=196449 … https://www.nature.com/articles/pcrj201372#Sec9 …pic.twitter.com/UnJ3TNif3y
Show this thread -
15/n I haven't gone through every study yet, but every one that I have is similarly worrying. I can't actually find a study that I'm not worried about in this analysis
Show this thread -
16/n If I go through the studies, I really can't find any that I would rate as a "low" risk of bias. Most of them are "high", some of them are really worryingly bad
Show this thread -
17/n Given this, I'm not sure the conclusions of the review make much sense at all. How can you interpret studies of this quality as moderate/strong evidence???pic.twitter.com/mvlxiUAvJX
Show this thread -
18/n I'm also worried about the heterogeneity of the 'usual care' groups across studies. Some of the usual care: - coffee - paracetamol - syrup - diphenhydramine - placebo - prednisolone These are very much not the same!
Show this thread -
19/n That's a big issue because you can't really combine the effect of coffee with diphenhydramine and expect it to make sense, but that's what the authors did
Show this thread -
20/n Ultimately, I think the only real conclusion you can draw here is that we have very little idea whether honey reduces symptoms for URTI/cough, and that the research is quite problematic
Show this thread -
21/n Not something that'll make headlines, perhaps, but sadly that's often how these things go
Show this thread -
22/n N.B. this study has already hit ~800 on Altmetric, been covered internationally, and made huge newspic.twitter.com/bj72yqShfA
Show this thread -
23/n Something else I didn't mention. Every study that I've looked at so far used a per-protocol analysis, which is a huge and worrying issue All of these should be at a high risk of attrition bias, yet none were rated as such. Most of them were green (low risk)pic.twitter.com/Y6sXlnkJHe
Show this thread -
24/n I've gotta say, for anyone teaching students about how finicky bias can be in systematic reviews, this is a beautiful example of getting it wrong
Show this thread -
25/n Somehow there are more issues here. This study was included in the risk of bias, but even though it assessed honey vs placebo/salbutamonl (and found no effect) it is not in any of the meta-analyses Very weirdpic.twitter.com/8AaMA85ERm
Show this thread -
26/n This is even weirder when you consider that the authors report excluding a study for not providing data So they exclude one study and report it, but another just...disappears? So strangepic.twitter.com/oyrp57yP1Y
Show this thread -
27/n Another included study using a per-protocol analysis. This was at least rated correctly as at a high risk of biaspic.twitter.com/t2mRMIq539
Show this thread -
28/n Another one. This study was rated at low risk of bias for most domains. Here's how they described their randomization and allocation concealment. What do you think?pic.twitter.com/1NM606xcRg
Show this thread -
29/n I'll give you a head-start - if they literally don't report ~how~ patients were randomized, by definition this should be unclear or high risk of bias for the domain of random sequence generation
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.