Each day, Pushshift is now ingesting, indexing and storing over half a billion social media objects. The eventual goal is to be able to process up to one billion objects daily for researchers to use to better understand how social media platforms influence society.
Conversation
Replying to
The scale you're at is astounding. My system (using a small portion of pushift's dataset) is absolutely chugging at only 8 billion objects 😰
1
2
Replying to
That's awesome! Really happy to hear people are benefiting from what we do.
1
1
Replying to
Are there plans to update the bulk reddit files at files.pushshift.io/reddit/ ? Looks like most/all of 2020 is missing and these are my main ingest source :)
1
3
Replying to
Yes. So to give some background, the last couple months have been very difficult since we have been moving our infrastructure into the cloud. We are going to start maintaining the monthly dumps again and get caught up as soon as possible. Realistically I would say that we
1
4
should get completely caught up by late spring of this year and then maintain them regularly from that point forward.
2
3
Replying to
Woah thats a big change and I'm sure quite a lot of work!
Your work is greatly appreciated as always
1
1
Replying to
Thanks! Feedback is always helpful -- especially knowing what parts of our service are the most used, etc.
Replying to
I (and my research partner) are probably an outlier compared to your usual users. We've made our own local database for the sake of throughput/weird queries and use the monthly dumps as our upstream data source on a 6+ month delay. We don't need 2020-21 data yet, so no rush!
2
1
Replying to
Just out of curiosity, what stuff are you using for your backend? Postgres? Elasticsearch? Are you mainly using Python?
1
1
Show replies