The Pushshift file repo files.pushshift.io is a bit slower than I'd like to see so we will be moving the data over to a server with NVMe storage over the next few weeks. The internet bandwidth is fine, but the hard drives are platter drives and when there are multiple
Conversation
downloads in progess, the random I/O of the drives causes a severe drop in IO bandwidth which lowers the overall download speeds tremendously. The new server will have over 500,000 IOP/s instead of just 250.
Replying to
I mentioned before, but you could try crowd sourcing the distribution via torrent. Then you would only need to seed. I'm sure there are many people in the community who would be willing to donate bandwidth.