There's going to be some brief API downtime today (~20 minutes) because I don't have time for frou-frou failover to the backup server; I have to replace some hard drives and then get the hell over the Sierra Nevada in a rental Mazda full of your data before the blizzard hits.
-
-
So the box in the photo above has 29 TB of user data. Pinboard has a 100 Mbps connection to the outside world; if I used 100% of that capacity, I could theoretically back up about 1 TB/day to a remote undisclosed location, so it would take about a month to move this data
Show this thread -
Except in practice, writing to the hard drives is slower than even that slow network connection, because they have to be in a configuration where if some of them fail, the data is not lost. So the whole process is like drinking a swimming pool through a straw.
Show this thread -
Over the years, we've made the pool way bigger, but the straw hasn't grown much. This holds true at every level—the CPU, the storage system, the data center. So 98% of modern programming is figuring out how to get around limits on moving ginormous amounts of data quickly
Show this thread -
Many smart people spend their careers on this. Some solutions include: being really smart about figuring out only what's changed, not the whole enchilada. Or copying stuff to multiple places. Or just paying a king's ransom for the biggest straw you can get.
Show this thread -
Or if you're not so smart, you can rent a fancy Mazda (that detects stop signs!) and drive your backups and your Japanese robot toilets through the basin and range in the snow. Half the cars on the road are full of hard disks and Japanese toilets. Look around you.
Show this thread
End of conversation
New conversation -
-
-
My God… it’s literally the “station wagon full of [hard drives] hurtling down the highway”
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
I can't help noticing that nobody seems surprised or puzzled by why you're traveling with one or two Japanese toilet attachments on car seats
-
Very few gas stations have one so you need to carry your own
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.