it helped a lot for me to learn about large deviations inequalities. you can pretty precisely quantify how unlikely it is for a large number of iid samples from a distribution to deviate significantly from the original distribution. it’s wacky stuff
for sure. it sounds like an innocent assumption but it's really not. actually quite hard to give a satisfying account of like, why we expect coin tosses or dice rolls to be even approximately iid