Conversation

Big news! Today we (

, @ncri?) have released a paper about the Pushshift Reddit Dataset! This is my first step into making a more direct contribution to science and I couldn’t be more excited to share. Paper: arxiv.org/pdf/2001.08435

101

269

JSON Baumgartner

@jasonbaumgartne

Jan 24, 2020

1/ While the paper includes a DOI that points to a sample of the data, it’s important to note that we make billions of posts and comments available via a searchable API via pushshift.io. Just looking at the volume of data shows how much Reddit has grown over time!

JSON Baumgartner

@jasonbaumgartne

2/ Releasing this paper is the culmination of 5 years of effort. I started Pushshift as a side project, and while I hoped that people would find it useful, I never imagined the kind of impact it would have. Over 100 peer reviewed papers have relied on Pushshift for data!

6:51 PM · Jan 24, 2020Twitter Web App

Replying to

3/ Pushshift is a community of researchers, data journalists, and citizen scientists. Over the years, we’ve built up an active subreddit, and fostered a collaborative Slack workspace, even building tools that others can use in their own collaborative groups.

4/ .

is currently coordinating a group of talented people building an easy to use app version of many of the tools we used in this academic research which they are sharing in beta today. -media-analysis-toolkit.herokuapp.com

JSON Baumgartner

@jasonbaumgartne

Jan 24, 2020

5/ Pushshift is not alone in its mission and service to the community. There are other great services that we took inspiration from, like

@media_cloud

, and they are invaluable resources for anyone that does data science.

JSON Baumgartner

@jasonbaumgartne

Jan 24, 2020

6/ In the end, what started as a side project has grown to be much more. This paper demonstrates our commitment to the community, both in continuing to provide the services that have pushed forward the frontier of science, but also our ongoing efforts to improve Pushshift!