Happy Friday! Today I will be dumping to the file repo (files.pushshift.io) the first quarter comments for 2020. It took a while to compress the data but considerably less time than previously with my new AMD 5950x processor (I swear I'm not plugging for AMD!) This weekend
Conversation
Replying to
I should be finishing up second and third quarter 2020 and by early next week I'll have 4th quarter and 1st quarter 2021. This will get the file repositories caught up for Reddit data for researchers.
1
9
The file format will be the same as previous files, however we are switching to a daily dump pipeline. I will also be providing a simple python script to make it easier for researchers to download bulk days from the file repositories.
1
11
The filenames will take the formats:
RC_YYYY-MM-DD for comments and
RS_YYYY-MM-DD for submissions.
Let me know if you have any questions!
4
9
This Tweet was deleted by the Tweet author. Learn more
He won't. I've sent him a DM and he hasn't responded. "Opt out" lmaooo
2
1
Show replies
Replying to
I'm not seeing any new Reddit uploads: are they publicly visible or is there another way of accessing them? Thanks!
Replying to
did these ever get uploaded? I don't see them anywhere on the file store
1
1
Replying to
They will be up shortly -- The main drive ran out of space so we have a larger SSD on order. We're putting the file repos on SSD because the spinny hard drive cannot access data fast enough and causes slow downloads. It should get here Friday so the data should be up this weekend
1