Conversation

Are you planning to add the latest Pushshift dumps to BigQuery? I'm already leaning in the direction of doing it myself since I don't have the required storage on my machine.

Quote Tweet

JSON Baumgartner

@jasonbaumgartne

Jun 26

Pushshift files.pushshift.io now points to the correct location for all recent data. From this moment onward, please use files.pushshift.io (it may take a few hours for the DNS to fully propagate for everyone, so you can use repo.pushshift.io until it does). Thx!

Replying to

and

It was getting adding back when Felipe was working at BQ but now that he has moved on, we'll have to get in touch with someone on their team to ask them if they can continue the tradition. You shouldn't have to pay for it.

5:46 AM · Jun 27, 2022Twitter Web App

Replying to

and

I forgot Felipe is now at Snowflake! I don't know whether there's such a thing as "public" Snowflakes? Especially since you've demoed loading PS dumps to it before!

medium.com

Loading 261GB of reddit comments into Snowflake

I used to load reddit comments onto BigQuery, now it’s time to upgrade my pipelines to Snowflake. Let’s get started with 261GB of reddit…

Replying to

and

I've been meaning to set up some time to talk to Felipe about this and to see if we can start trying out snowflake. I guess if he sees this, now would be a great time to check with him. We have over 250,000 unique visitors per month using Pushshift so that woiuld be a ton of

Show replies