The Pushshift file repo files.pushshift.io is a bit slower than I'd like to see so we will be moving the data over to a server with NVMe storage over the next few weeks. The internet bandwidth is fine, but the hard drives are platter drives and when there are multiple
Conversation
Replying to
downloads in progess, the random I/O of the drives causes a severe drop in IO bandwidth which lowers the overall download speeds tremendously. The new server will have over 500,000 IOP/s instead of just 250.
1
5