if you have 1.3TB of space and a good pipe to what I think is US-EAST-1 then here you are, Strava archiver https://gist.github.com/whitequark/cd89594b797217992d8263325cb77d75 …https://twitter.com/whitequark/status/957744553976528909 …
-
Show this thread
-
I am now going to port the archiver to use EC2/S3 in an attempt to get a fatter pipe, but I'll keep my existing job running
3 replies 1 retweet 17 likesShow this thread -
-
optimized EC2 task downloading at 180Mbps peak
2 replies 0 retweets 9 likesShow this thread -
switched to an m5.x2large; 250 Mbps sustained, 350 Mbps peak
2 replies 0 retweets 8 likesShow this thread -
400 Mbps sustained, 780 Mbps peak the scraper is written in Ruby and uses blocking I/O by the way. never tell me how Ruby or blocking I/O cannot scale, ever again
5 replies 16 retweets 72 likesShow this thread
Replying to @whitequark
Blocking IO is almost always the simplest & most efficient solution. So tired of the singleton event loop snake oil.
1 reply
1 retweet
1 like
-
-
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.