The research team that is given this list of ids then need to "rehydrate" the tweets via Twitter's API. One of the things that I am looking into is what metadata can be shared along with each tweet id. This would enable researchers using the data to filter out tweets that aren't
Conversation
needed for their research and reduce the number of API requests needed to rehydrate the remaining tweet ids. After I confirm with Twitter if sharing metadata for each tweet id is possible, I wanted to ask other researchers which pieces of metadata are most important.
1
7
Right now, I am creating a list of metadata fields to accompany each tweet id. Currently, I have the following metadata fields in mind. My question to other researchers is what other metadata would be helpful to aid in pre-filtering the data. Here are the fields I've added so far
1
6
tweet id (bigint)
user_id (bigint)
conversation_id (bigint)
is_retweet (bool)
is_quoted_tweet (bool)
is_reply (bool)
is_root (bool)
last_update_time (int)
retweet_count (int)
favorites_count (int)
reply_count (int)
contains_location_data (bool)
contains_media_types (enum int)
1
6
is_verified (bool)
user_follower_count (int)
user_creation_epoch (int)
Keep in mind that some fields may not be allowed via Twitters TOS (I need to clarify this with Twitter). Again, the main objective is to give as much metadata as possible to assist researchers in easily
1
5
pre-filtering the data before rehydrating tweets. If you can think of any other useful metadata to accompany each tweet id, please feel free to add your ideas and recommendations.
I will update with more info once I get more guidance from Twitter.
7
8
Replying to
how about instead of providing the metadata with tweet IDs, you provide a queryable API filtering on the metadata on your end? you can share only tweet IDs from the resulting query made to said API. this wouldn't require an approval from Twitter either since no metadata is shared
2
1
2
Oh wow, that's actually a very good idea and I think it would comply with the ToS and the developer agreement of Twitter. I think one could even add a filter for words in the tweets or user bios!
1
2
precisely. essentially, this would be the Twitter search endpoint simulated. The only drawback being that it returns just the `tweet_id` attribute instead of the entire Tweet Object.
1
also, in case Twitter does permit the sharing of metadata, and you choose to go ahead with that instead, `created_at` and `lang` are two crucial attributes that I found missing in your list.
2
1
I agree lang is essential. I would probably create an int8 mapping to lang to reserve one byte for that field and then provide a translation script with the metadata.