I just want to take a moment to thank everyone who used the Pushshift suite of APIs for their patience. There were growing pains associated with the original infrastructure and it necessitated completely revamping everything from the ground up within AWS.
Although it will cost
Conversation
more to run everything from the cloud, we do have runway capital to see us through for the next 9-12 months while we start working on getting researchers and universities onboard.
What I can tell you is that the new infrastructure will be much better than the existing one and
1
9
we will have the capability to expand by adding more nodes as needed. The projected cost for the new infrastructure will be around $15-$20k per month and will enable us to quadruple the size of our Elasticsearch cluster along with adding much improved ingest pipelines and
1
5
monitoring scripts. That means no more falling behind by hours on our ingests for Reddit and other platforms. Also, latency should be much improved for existing API calls. We will be scaling up our storage to around 5 petabytes in total (much of which will be high speed NVMe
1
8
drives. This will allow us to generate datasets for any research project in a matter of days instead of weeks or longer. We are still on target to launch the new infrastructure by mid-February at the latest -- with the new ingest pipelines coming online in the next week.
Show additional replies, including those that may contain offensive content
Show