The first thing I do when I see a headline like this is try to find a press release. It gives you a lot of insight into how the story came about This is what I foundhttps://www.medpagetoday.com/meetingcoverage/ddw/79919 …
-
-
For the epi nerds, when I run it with a fixed-effects model my results are the same as those reported in the paper, but my random-effects model CI crosses 0
Show this thread -
But now comes the interesting part - what happens if I take out that single paper that appears to be driving the result? What do you reckon?
Show this thread -
Here's the result. The association disappears completely It looks like one study is driving all of these resultspic.twitter.com/hcmXJePEol
Show this thread -
So what is this study? Essentially, a simple observational survey of people in Singaporepic.twitter.com/H2qxbLvGJD
Show this thread -
Now, I'm not going to critique this piece of research in-depth, but I think it's worth noting that it only surveyed 300 people, of whom 80 had IBS The other studies looked at a total of ~2,500 people
Show this thread -
So what we're seeing in the meta-analysis is basically a series of negative results being totally overset by a single positive result That is not great scientifically!pic.twitter.com/xv5a3lDScx
Show this thread -
It's a bit like tossing a coin 5 times, getting 4 tails and 1 heads, and concluding that heads is the right answer
Show this thread -
This is especially true when you consider that the p-value is 0.064, which means that these results aren't even ~technically~ significant in any model!
Show this thread -
But bringing this back to
#scicomm - how is a journalist meant to know this? It's complex stuff. Most scientists I know aren't comfortable re-running a meta-analysis to see what happens when you exclude studiesShow this thread -
And the press release, let's remember, is astonishingly positive. No mention of the MASSIVE question mark remaining after this research, just "pet owners more likely to have IBS"
Show this thread -
The real finding from this analysis is that there may be a very modest increase in risk of IBS from owning a pet, but this seems unlikely at present based on the totality of the evidence
Show this thread -
Who do we blame for the misreporting? I'll leave that to you There are many steps along the way that could've corrected this, but none were taken
Show this thread -
SMALL CORRECTION The forest plot I included earlier in the analysis of the random-effects model was from the log-transformed variables (oops) here's the plot once exponentiated:pic.twitter.com/Me4zJfoI6w
Show this thread -
Also, the p-value is 0.064 for this model, which is technically not significant. The effect size is also different from that reported in the abstract, however if I run a fixed effects model everything is exactly the same so I suspect that's what was actually done here
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.