Pushshift is completely rewriting the Reddit ingest script to handle parallel workers. What does this mean for you? No more large delays in updating comment and submission data!
I'm hoping to have this completed in the next few days.
Conversation
Replying to
I dont know if now is the time, but it would be super useful to understand more about how the pipeline works if you have a chance to write about that. I think it's especially important for research based on PushShift's data. PS. I'd be happy to help if you want any.
1
2
Replying to
Good point. I'll probably open-source the new ingest methodology so I can get input from other researchers and make sure the methods used are sound and efficient.
1
2
Show replies