Conversation

I am creating a project board and eventually a wiki for the Google #bigquery / Pushshift integration. However, #reddit comments are now streaming live into BQ! Could someone please confirm that they can access the dataset?

github.com

GitHub - pushshift/google_bigquery

Contribute to pushshift/google_bigquery development by creating an account on GitHub.

JSON Baumgartner

@jasonbaumgartne

May 23, 2018

You will need to switch from legacy SQL on the Google BQ dashboard. Just try to do a simple select: SELECT * FROM `pushshift.rt_reddit.comments` LIMIT 10 The dashboard should be located here: bigquery.cloud.google.com

Replying to

"Query complete (3.4s elapsed, 0 B processed)" looks good

Replying to

Awesome! You're able to access it. Thanks so much!!!

Replying to

New ones coming in, too, judging by "order by desc on created_utc"

Replying to

That's great news. The feed should be very close to real-time (a 1-3 second delay at most usually).

Replying to

So is the idea to provide recent comments only, or will you be pulling in the backlog too and letting people utilise partitions?

Replying to

I am starting the real-time ingest now so going forward, people can see what's going on with Reddit on an hourly or daily basic (the table is partitioned by day). There is also the monthly data tables that include more accurate scores (since they have had time to settle)

Replying to

and

This isn't meant for historical research -- more along the lines of hourly or daily recaps and analysis going forward.

1:54 AM · May 23, 2018Twitter Web Client

Replying to

Why would I use this over, say, the PushShift API? Because I prefer BQ?

Replying to

#bigquery would be far faster if you wanted to do complicated analysis involving regex, etc. With this system, you could very quickly find all comments with a very specific regex (something the API lacks in support). So this provides a lot more power in certain areas.

Show replies