Conversation

JSON Baumgartner

@jasonbaumgartne

The Pushshift Twitter archives now contain over 150 billion tweets (1.3 petabytes of data).

7:24 PM · Jan 9, 2021Twitter Web App

Replying to

Is it also archiving video, pictures and other media on Twitter?

Replying to

No -- too much data right now to archive all the media.

Replying to

did you opensource the technique you discovered to do unlimited tweet downloads from twitter without key? I found the other technique you open-sourced:

github.com

GitHub - pushshift/token_manager: Code to handle multiple Twitter user access tokens when making...

Code to handle multiple Twitter user access tokens when making requests - GitHub - pushshift/token_manager: Code to handle multiple Twitter user access tokens when making requests

Replying to

and

Nevermind, found a library that must use the same technique: pypi.org/project/twint/

Replying to

Holy moly!! Thank you for all the valued work!! That seems expensive. Sending a quick donation now

Replying to

Wow... Are these archives available somewhere?

Replying to

No (due to Twitter TOS issues)

Replying to

Have you got decent docs or tweets on the infrastructure Pushshift runs on? I'm always impressed when I see but can't locate them :)