There's going to be some brief API downtime today (~20 minutes) because I don't have time for frou-frou failover to the backup server; I have to replace some hard drives and then get the hell over the Sierra Nevada in a rental Mazda full of your data before the blizzard hits.
-
-
For people wondering why I'm making mid-pandemic road trips with big boxes of disks, the problem boils down to an oddity of progress in tech. You can now store absurd amounts of data cheaply, but it's still hard to move it around in bulk (both inside and outside the computer)
Show this thread -
So the box in the photo above has 29 TB of user data. Pinboard has a 100 Mbps connection to the outside world; if I used 100% of that capacity, I could theoretically back up about 1 TB/day to a remote undisclosed location, so it would take about a month to move this data
Show this thread -
Except in practice, writing to the hard drives is slower than even that slow network connection, because they have to be in a configuration where if some of them fail, the data is not lost. So the whole process is like drinking a swimming pool through a straw.
Show this thread -
Over the years, we've made the pool way bigger, but the straw hasn't grown much. This holds true at every level—the CPU, the storage system, the data center. So 98% of modern programming is figuring out how to get around limits on moving ginormous amounts of data quickly
Show this thread -
Many smart people spend their careers on this. Some solutions include: being really smart about figuring out only what's changed, not the whole enchilada. Or copying stuff to multiple places. Or just paying a king's ransom for the biggest straw you can get.
Show this thread -
Or if you're not so smart, you can rent a fancy Mazda (that detects stop signs!) and drive your backups and your Japanese robot toilets through the basin and range in the snow. Half the cars on the road are full of hard disks and Japanese toilets. Look around you.
Show this thread
End of conversation
New conversation -
-
-
Where’s the data going to end up? When I worked at Box, we used Amazon Snowball to forklift huge, multi-TB and PB data sets. It might be overkill for your situation and I’m sure there are some caveats, but for some scenarios it was a godsend.
-
The data eventually lives under my desk, a thousand miles away from the servers it backs up. I can look at Amazon pricing again, but it made no sense when I evaluated it a couple of years back
End of conversation
New conversation -
-
-
This Tweet is unavailable.
-
So like, if I do the same thing I do now, but pay Amazon?
End of conversation
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.