Conversation

I am creating a project board and eventually a wiki for the Google #bigquery / Pushshift integration. However, #reddit comments are now streaming live into BQ! Could someone please confirm that they can access the dataset?

github.com

GitHub - pushshift/google_bigquery

Contribute to pushshift/google_bigquery development by creating an account on GitHub.

JSON Baumgartner

@jasonbaumgartne

May 23, 2018

You will need to switch from legacy SQL on the Google BQ dashboard. Just try to do a simple select: SELECT * FROM `pushshift.rt_reddit.comments` LIMIT 10 The dashboard should be located here: bigquery.cloud.google.com

Replying to

"Query complete (3.4s elapsed, 0 B processed)" looks good

Replying to

Awesome! You're able to access it. Thanks so much!!!

Replying to

New ones coming in, too, judging by "order by desc on created_utc"

JSON Baumgartner

@jasonbaumgartne

Replying to

@flashman

That's great news. The feed should be very close to real-time (a 1-3 second delay at most usually).

1:48 AM · May 23, 2018Twitter Web Client

Replying to

So is the idea to provide recent comments only, or will you be pulling in the backlog too and letting people utilise partitions?

Replying to

I am starting the real-time ingest now so going forward, people can see what's going on with Reddit on an hourly or daily basic (the table is partitioned by day). There is also the monthly data tables that include more accurate scores (since they have had time to settle)

Show replies