2/n The updated paper is here https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2772834 …
And you can read @ikashnitsky and my original commentary on the paper herehttps://osf.io/9yqxw/
-
-
Show this thread
-
2.5/n Important to note that this is a very influential paper. It has been in >100 news stories, and has been cited by the EU and WHO (!) Worrying that until recently it was openly wrong
Show this thread -
3/n In brief, the original paper made a number of obvious errors in analysis, and conducted a largely meaningless analysis using years of life lost (YLL) to compare the harms of COVID-19 and school closures
Show this thread -
4/n So, what's changed? Well, the main structure of the study is the same. They've taken an estimate of the impact of school closures from Argentinian children in the 70s, and applied it directly to children learning from home during the COVID-19 pandemic
Show this thread -
5/n Previously, this estimate entirely relied on an incorrect meta-analysis that showed that every extra year of schooling reduced your risk of death by 25% over your entire lifespan This has been removedpic.twitter.com/gNMoGSb6eu
Show this thread -
6/n (If I was of a skeptical bent, I'd guess that this meta-analysis was removed instead of corrected because simply correcting the model entirely reverses the main findings of the paper)
Show this thread -
7/n Instead, the authors have completely arbitrarily divided their analysis into two sections - that based on 'US' and 'European' studies This gives them some...interesting resultspic.twitter.com/8zeTqcFq3P
Show this thread -
8/n If we take these results at face value, closing schools during COVID-19 in the US either killed ~10x more or 5x less people than had they stayed open, depending on whether the analysis is based on 'US' or 'European' studies
Show this thread -
9/n So, what are these studies? Well, here's the first obvious error that the authors have left in. The two US studies you can see in the table from the supplementaries belowpic.twitter.com/a6S6KFO3b1
Show this thread -
10/n Problem is, Mazumder (2008) is simply a re-examination of THE SAME DATA (1919 and 1939 schooling law changes in the US) as Lleras-Muney (2005) It's simply incorrect to just blindly chuck these estimates into the same model as if they aren't intrinsically relatedpic.twitter.com/YTelTNbBDi
Show this thread -
11/n And it gets worse. As we pointed out in our critique, the MAIN CONCLUSION from the Mazumder (2008) paper was that SCHOOLING HAD MINIMAL OR NO IMPACT ON MORTALITY So how did the authors instead estimate a 35.2% reduction in relative risk????pic.twitter.com/arDt2wf6te
Show this thread -
12/n It appears that the authors have cherry-picked a single result from one table of the paper that was actually just recreating Lleras-Muney's results from the original paper using the second methodological approach, rather than using the actual results from the paperpic.twitter.com/1G4F0cylaa
Show this thread -
13/n It gets worse. In 2010, Mazumder published an erratum to the paper which I will quote here: "there is little compelling evidence suggesting a causal link between education and mortality based on Census data and compulsory schooling laws"pic.twitter.com/d22AgEPntK
Show this thread -
14/n As ever, I am not here to litigate intentions, but it is amazingly strange to pick the wrong number from a table of a paper that flatly contradicts your main assertion and then lump it in with the paper ~that it was correcting~ as if this wasn't scientifically flawed
Show this thread -
15/n I mean, using a paper that says "there is no relationship between missed schooling and mortality" to support your argument that there is a relationship between schooling and mortality is...something
Show this thread -
16/n But this brings us back to the results. Either missed days of schooling cost 0.8 (0.1-2.4) million YLL or 13.8 (2.5-42.1) million YLL Why the massive divergence?pic.twitter.com/RgPgiyDoRY
Show this thread -
17/n Well, if you use the incorrect estimates from Mazumder/Lleras-Muney, you get an average of a 45% reduction in RELATIVE RISK OF DEATH AT EVERY AGE per year of schooling This is hilariously implausiblepic.twitter.com/Mt8CSd9t4T
Show this thread -
18/n For context, this would mean that adding 3 years of schooling for every child would reduce their risk of death to literally 0% for the rest of their lives, effectively rendering them immortal
Show this thread -
19/n The authors justify this with some dubious language about how US studies are more similar than Nordic ones, despite the fact that the US studies are based on legal changes in 1919 and 1939, at which time the US was (for example) still segregatedpic.twitter.com/5aLTJeV05C
Show this thread -
20/n It is also very strange that the authors are happy to use evidence from an Argentinian study on children markedly different from those in the US but balk at studies on European children It seems an obvious contradiction
Show this thread -
21/n Nevertheless, what we now have is a study that, weirdly, says that either school closures have cost very few YLL or a wildly absurd overestimate, depending on whether you limit your analysis to only the studies that the authors prefer arbitrarily to usepic.twitter.com/jCRTXAJtzq
Show this thread -
22/n Many of the other obvious flaws in the paper remain uncorrected, and I'd urge you to read our full critique if you're interested:https://osf.io/9yqxw/
Show this thread -
23/n That being said, it is worth saying that the authors and journal have at least taken ~some~ action here, and correcting the mathematically impossible model is a good first step for this paper
Show this thread -
24/n My hope is that now we can further correct the other obvious mistakes and issues, and come to a more realistic estimation, because this paper is currently impacting policy on an international scale And it is still simply wrong
Show this thread -
25/n I also think it's worth noting the process it took to get this paper corrected even this far The original response from the lead author and journal editor was, to quote exactly, "you are not right just because you think you are"
Show this thread -
26/n This was after
@ikashnitsky and I pointed out that the paper was MATHEMATICALLY IMPOSSIBLE Quite a worrying way to respondShow this thread -
27/n After we published our preprint critique, and it was reported on in the Guardian, we were told to submit a comment on the piece as soon as possible online and they'd get back to ushttps://www.theguardian.com/science/2020/dec/08/coronavirus-study-that-found-us-school-closures-cut-life-expectancy-criticised-by-epidemiologist …
Show this thread -
28/n Two months after the initial emails, and over a month after we submitted the comment, we have this correction published Unfortunately, the study has already had an enormous impact, and changed lives across the world
Show this thread -
29/n You will probably be interested
@stephaniemlee@MelissaLDavey@apsmunro@DrZoeHyde@devisridhar@DFismanShow this thread -
30/n Overall, what we have is a paper where the mathematically impossible results have been removed, but is still flawed in numerous ways and useless as evidence for decision-making
Show this thread - Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.