Conversation

Can I get some feedback from people that use Pushshift Reddit data -- should Reddit dumps go into daily files or should I keep the monthly format? There are pros and cons to both but I wanted to see what the community thought.

Keep Reddit dumps monthly
57.1%
Switch to daily dumps
42.9%

91 votesFinal results

Replying to

Monthly keeps them more manageable for me. I think it is easier to drill down to daily trends as is.

Replying to

The one big benefit of daily dumps is that the daily dump when compressed would generally fall beneath 512MB in size which means Cloudflare would cache the files making them much faster to download (if they remain in cache which they should)

Replying to

Ah. Did not know that. That would also be helpful. Hmm... makes my vote is not as clear!

JSON Baumgartner

@jasonbaumgartne

Replying to

@TawfiqAmmari

I see your point about ease of download but if I wrote a Python script to collect daily files for a specific time range, maybe that would be a good compromise?

11:51 PM · Jun 3, 2021Twitter Web App

Replying to

Yup. That would do it. I would do just that as well. In the grand scheme of things, it would mean a little change to my code.