2/n Paper is here. It's actually very simple, basically the authors took an estimate of how many days of school kids lost due to closures during COVID-19, an estimate of how much that impacts their years of life, and multiplied outhttps://jamanetwork.com/journals/jamanetworkopen/fullarticle/2772834 …
-
-
Show this thread
-
3/n Specifically, they said: 1. Children miss days of school 2. Days of school cause decreased educational attainment 3. Decreased educational attainment causes less lifepic.twitter.com/oeS65P1FWl
Show this thread -
4/n They then compared this with estimates of the number of years of life lost due to COVID-19 already, and found that the loss of educational attainment caused more death (with 98.1% probability!)pic.twitter.com/ThAbSAERqa
Show this thread -
5/n Now, that sounds very robust. 98.1% probability is extremely certain! It's also, as far as I can tell, an incredibly meaningless number. NONE OF THIS IS THE SLIGHTEST BIT CERTAINpic.twitter.com/DV4F2rHDFu
Show this thread -
6/n Firstly, the paper that all of this extrapolation is based on - reference 14. This is where the authors got the relationship between days of schooling missed and overall educational attainmentpic.twitter.com/aeCYm7YNnZ
Show this thread -
7/n This is a long and fascinating examination of the association between teachers' strikes in Argentina and the eventual earnings that children achieved when they grew up
Show this thread -
8/n Note there that I said ASSOCIATION. This is a key point. The authors of this paper WARN AGAINST USING IT CAUSALLY because, among other issues, the data for children in Argentina in the 70s and 80s (especially schoolkids) was not greatpic.twitter.com/2GSkNtgLDy
Show this thread -
9/n Indeed, the main analysis of this paper explicitly excludes the impact of missing secondary education because their data was incomplete and many Argentinian children in the cohort at the time did not attend secondary schoolpic.twitter.com/qO0MHdrU1w
Show this thread -
10/n It should also be noted that the impact of TEACHERS' STRIKES on ARGENTINIAN CHILDREN may not mean much when we're talking about American children, some % of whom could learn at home and thus were not explicitly missing class
Show this thread -
10.5/n (Also, whether the impact of teachers formally striking on children in the 70s is the same as missing days of in-person education during a pandemic in 2020...I would say probably not)
Show this thread -
11/n Next, the authors did something odd. They decided to compare this to the DIRECT mortality from COVID-19. This means that they are only counting people who have died as a direct result of the diseasepic.twitter.com/MQiG2PapOc
Show this thread -
12/n But, if we're comparing the total impact of missed education on children over a lifetime, this makes no sense. COVID-19 has many other harms, after all, some of which are disputed but many of which are easily attributable
Show this thread -
13/n Where are the years of lost life due to ICU stays? Hospitalizations? Disability, mental health issues, avoided care due to full hospitals, etc etc etc???
Show this thread -
14/n (This is also a problem for the YLL calculation for children missing school, FYI, it is FAR more complex than just missing educational attainment. So many potential things that are not included here!)
Show this thread -
15/n Ultimately, what we get in the end - the "98.1% probability" figure - makes almost no sense at all. Even if the Argentinian study was applicable to the US situation, which it clearly isn't, the counterfactual is just inappropriate
Show this thread -
16/n It may be possible to do a rigorous cost/benefit of school closures, but I really don't see how this can be considered one
Show this thread -
17/n At best, what we've got here is a very rough inference about what the impact of school closures could possibly have been (as long as US kids in 2020 are the same as Argentinians in 1977), compared to an undercount of the impact of what COVID-19 probably was
Show this thread -
18/n I should add - I am not saying that closing schools is necessarily a positive in this thread. What I AM saying, however, is that we could not possibly know whether it was or not from this paper
Show this thread -
19/n Many further criticisms of this paper that I missed in this also excellent thread. It appears that almost none of the assumptions in the paper make any sense at allhttps://twitter.com/ikashnitsky/status/1328121853970436100?s=20 …
Show this thread -
20/n Another point I've been thinking about - the authors fail to consider the impact of an ongoing epidemic on school attendance. Lots of cases in the local area probably impacts schooling as well!
Show this thread -
21/n Also, what's the impact of parents/carers getting sick and dying? Presumably this will have an outsized impact on the children it affects, which would cause (based on this model) large increases in YLL
Show this thread -
22/n Ok, there's another HUGE problem with the study The findings are entirely based on the idea that an extra year of schooling reduces your risk of death by 25%, a figure reached by aggregating studies in a meta-analysis modelpic.twitter.com/va8PWXeBc3
Show this thread -
23/n This number is the ENTIRE BASIS for the paper. If schooling reduces mortality by less than 25%, then the number of deaths caused by missed school days will drop as well And we have a problem, because this 25% figure is...wrong
Show this thread -
24/n Firstly, we have these top 2 papers. Because they're from the US, they are more heavily weighted than the others (this is arbitrary, but whatever) Except the values for the second paper are flatly incorrectpic.twitter.com/VxC2Krkj3z
Show this thread -
25/n Mazumder (2008) did not find a relative risk of 0.65 (0-1.3), it found an RR of 0.89 (0.74-1.04) The paper is here. I've read it very carefully and this appears to be the main finding from the author https://www.chicagofed.org/publications/economic-perspectives/2008/2qtr2008-part1-mazumder …pic.twitter.com/1FY15LCXFe
Show this thread -
26/n So the number is wrong, but this brings us to another issue - the two papers at the top that were double-weighted...are ON THE SAME DATASET This is a massive issuepic.twitter.com/trCaHySTYE
Show this thread -
27/n You see, Lleras-Muney compiled an estimate based on a number of factors on the reduction in risk from 1 year of extra schooling. Mazumder, three years later, took that same database and re-analyzed it considering additional factors, and estimated a lower reductionpic.twitter.com/2pWUNsP6Qb
Show this thread -
28/n However you slice it, it's inappropriate to just bung these two estimates together into a model and treat them as separate. I would argue that the Mazumder paper is probably a better estimate, but either way what they've done here is wrong
Show this thread -
29/n If I re-run the meta-analysis they've purported to use with the correct numbers, we end up with this graph instead. A relative reduction in risk of death of 5% for every year of schooling This is A QUARTER of the estimate used in the studypic.twitter.com/xQk5F1tDzT
Show this thread -
30/n So AT BEST assuming there are no other errors, the estimate of the years of life lost due to school closures should be divided by 5
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.
post-pub peer-review THREAD
1/