In order to share Twitter data with academic researchers, it is important that Pushshift does so in a way that complies with Twitter's TOS.
Recently, Twitter made some modifications to its TOS which allow for sharing an unlimited number of tweet ids with other researchers.
Conversation
The research team that is given this list of ids then need to "rehydrate" the tweets via Twitter's API. One of the things that I am looking into is what metadata can be shared along with each tweet id. This would enable researchers using the data to filter out tweets that aren't
1
8
needed for their research and reduce the number of API requests needed to rehydrate the remaining tweet ids. After I confirm with Twitter if sharing metadata for each tweet id is possible, I wanted to ask other researchers which pieces of metadata are most important.
1
7
Right now, I am creating a list of metadata fields to accompany each tweet id. Currently, I have the following metadata fields in mind. My question to other researchers is what other metadata would be helpful to aid in pre-filtering the data. Here are the fields I've added so far
1
6
tweet id (bigint)
user_id (bigint)
conversation_id (bigint)
is_retweet (bool)
is_quoted_tweet (bool)
is_reply (bool)
is_root (bool)
last_update_time (int)
retweet_count (int)
favorites_count (int)
reply_count (int)
contains_location_data (bool)
contains_media_types (enum int)
1
6
is_verified (bool)
user_follower_count (int)
user_creation_epoch (int)
Keep in mind that some fields may not be allowed via Twitters TOS (I need to clarify this with Twitter). Again, the main objective is to give as much metadata as possible to assist researchers in easily
1
5
pre-filtering the data before rehydrating tweets. If you can think of any other useful metadata to accompany each tweet id, please feel free to add your ideas and recommendations.
I will update with more info once I get more guidance from Twitter.
7
8
Replying to
how about instead of providing the metadata with tweet IDs, you provide a queryable API filtering on the metadata on your end? you can share only tweet IDs from the resulting query made to said API. this wouldn't require an approval from Twitter either since no metadata is shared
2
1
2
Replying to
This is an amazingly good idea! I can't see why Twitter would have an issue with it unless it exposed contents of deleted tweets in some way (like querying an exact text piece and getting one tweet id / user id back from it).
I really hope Twitter would be open to this idea.
1
2
Replying to
that's a valid argument. i have to admit i didn't consider that. a possible workaround could be restricting queries on text to n (=3?) word sequences? one could theoretically still make multiple queries and get the tweet, but it'll now come with a lot more effort
1
1
Replying to
Yeah I like that idea. At the very least, one could put in a series of terms they want to use to extract tweet ids (e.g. riot, riots, protests, protest, protesters, etc.).
Replying to
yup, as i said, pretty much like Twitter's search endpoint. ideally, `q=['riot', 'protest']` should be able to fetch `tweet_id`s for tweets containing any of those words (simulating the word in sentence query for a list of words). alternatively, `q=['riot OR protest']` should too
1
you might want to check these out for ideas and reference on what can be achieved:
developer.twitter.com/en/docs/tweets
1
Replying to
additionally, I'd also recommend restricting queries being made based on both text (`q`) and `user_id` in the same query. that should further curtail efforts seeking to isolate deleted tweets. i can't think of something that would require one to make a query with both together.