2/n The paper is here, with that infamous graphic https://jamanetwork.com/journals/jama/fullarticle/2768533 …
-
-
Show this thread
-
3/n What did the authors do? Well, it was a really very simple paper (published as a research letter, so <1,000 words!) They looked at how many workers tested positive for COVID-19 in hospitals before and after mask wearing became mandatory
Show this thread -
4/n Prior to the mandatory mask-wearing, the percent positive - i.e. the percent of people who tested positive to COVID-19 out of those who were tested - appeared to be increasing After mask-wearing, it appeared to decrease instead Conclusion - masks work!pic.twitter.com/7xVyjsd1Np
Show this thread -
5/n But, some things immediately jump out when you read the paper that make it...worrying Firstly, the denominator is really weird. They divided by HCWs who tested positive that day+HCWs who NEVER tested positive...any daypic.twitter.com/sHtCWxz9aW
Show this thread -
6/n But presumably the number of HCWs who NEVER tested positive ANY day would always increase, unless the number of tests is fixed (which it presumably isn't) This would then mean that, over time, the % positive should decrease naturally
Show this thread -
7/n Then, another snag. The authors have drawn some lines on a graph, but not provided any statistical tests to see if these lines mean anything They report USING the R^2 value, but don't actually write it down! Not idealpic.twitter.com/UNjOmPs3Me
Show this thread -
8/n This XKCD comic makes this problem clear very elegantlypic.twitter.com/a5NGp47iPB
Show this thread -
9/n On top of all this, the authors report that the case number for COVID-19 continued to increase in Massachusetts (where the study was done), but don't discuss the PERCENT POSITIVE which is what this study is looking at
Show this thread -
10/n This is an issue, because if the state has similar trends in % positive as the hospital, then it's likely that what's seen in this study is simply reflective of the COVID-19 epidemic more broadly
Show this thread -
11/n Here are the % positive graphs side-by-side They look, to me, remarkably similar Maybe it wasn't the masks after all...pic.twitter.com/Rt9BVZL4eW
Show this thread -
11/n In addition, there are some other worries with the paper Because it's a research letter, they can't really go into confounding, but they do mention it as an issuepic.twitter.com/PkfzwYQOdE
Show this thread -
12/n Problem is, confounding in this study isn't just a small problem - it's an enormous, gaping hole in the reliability of the results
Show this thread -
13/n What if people were wearing masks before the study started? What if the population of HCWs changed during the pandemic? What if reporting of tests changed? What if they simply DID MORE TESTS? So many problems, no answers at all
Show this thread -
14/n To some extent, it's hard to blame the authors for these issues, but it shows why you probably shouldn't report on research letters - they just don't have enough information to make a balanced judgement
Show this thread -
15/n At the absolute minimum for this study, you need an adequate control group to even begin to understand what these results might mean Without that, about all we can say is that they made a pretty graph with lines on it
Show this thread -
Note: all of this is independent of whether you like masks! I'm not saying masks don't work, just that this study says nothing about the question
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.