There's going to be some brief API downtime today (~20 minutes) because I don't have time for frou-frou failover to the backup server; I have to replace some hard drives and then get the hell over the Sierra Nevada in a rental Mazda full of your data before the blizzard hits.
-
-
Except in practice, writing to the hard drives is slower than even that slow network connection, because they have to be in a configuration where if some of them fail, the data is not lost. So the whole process is like drinking a swimming pool through a straw.
Show this thread -
Over the years, we've made the pool way bigger, but the straw hasn't grown much. This holds true at every level—the CPU, the storage system, the data center. So 98% of modern programming is figuring out how to get around limits on moving ginormous amounts of data quickly
Show this thread -
Many smart people spend their careers on this. Some solutions include: being really smart about figuring out only what's changed, not the whole enchilada. Or copying stuff to multiple places. Or just paying a king's ransom for the biggest straw you can get.
Show this thread -
Or if you're not so smart, you can rent a fancy Mazda (that detects stop signs!) and drive your backups and your Japanese robot toilets through the basin and range in the snow. Half the cars on the road are full of hard disks and Japanese toilets. Look around you.
Show this thread
End of conversation
New conversation -
-
-
that seems pretty slow for 2021... if you had a 1Gbps (or greater) internet connection wouldn't network backups be a lot faster/more achievable at your scale?
-
This Tweet is unavailable.
- Show replies
New conversation -
-
-
Never had to deal with this amount of data myself, but only moving the difference using eg rsync wouldn’t work?
-
It's a big complexity tradeoff. Do you just do the whole thing on site from time to time with 100% assurance you got all the data, or do write a bunch of code to keep things in sync and throttled and so on? Not obvious which answer is right
End of conversation
New conversation -
-
-
So, uh, tell us you have winter tires? I would also suggest, I dunno, getting a cheapo baremetal server somewhere like he or ovh or something to mirror to.
-
I already host the site on cheapo bare metal servers. What's the attraction of adding more? The constraint is bandwidth to move the data around, not storage. And there's a real advantage to having a backup that's not online where mean people could hack into it and delete it
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.