The new #reddit ingest code appears to be stable. I've made some additional tweaks to improve the latency. Attached is a picture showing the latency (in seconds) and the percentage of comments retrieved within that time. A majority of comments are fetched within 5 seconds or less
Conversation
Replying to
I will be moving the new code to beta.pushshift.io so that others can help test. I anticipate the new ingest code will make it into production within the next couple of weeks.
I'll let you know when beta.pushshift.io is up and available for testing.
5
Replying to
Do you have a comparison image for the previous implementation? Are you using python?
1
1
Replying to
Yes I am using Python. The previous implementation was consistently behind by hours so it's a huge leap forward. Unfortunately, I don't have any stats right now for the current production implementation.
1