This is a rough rough draft that will eventually become a blog post -- but I wanted to share this now for comments / suggestions.
This Tweet was deleted by the Tweet author. Learn more
Conversation
Replying to
Oh wow! Never thought the snowflake parts would be possible to enumerate like that to make it possible to check IDs! This is a great idea!
1
1
Replying to
Can't see your code but immediately what came to mind is spreading out machine ids across API calls so you stop guessing early for ones that have low seq numbers, might speed things up if I understood correctly?
1
Replying to
Most of the time, they won't get above sequence id 0 -- I don't know how sequence ids are allocated but I suspect it would be sequential for each server. For instance, machine id #123 happens to process two tweets in the same millisecond, the first will get 0 and the next 1
1
Replying to
Yep, that's it - wondering if it's worth checking lots of seq. 0 IDs first, then move on to seq. 1 in next batch, given that lookup/ids API call tells you the non existing IDs, you can skip / move on and generate the next millisecond's IDs to keep API calls efficient.
2
Replying to
This is a rough draft and I'm sure it can be improved, but at least this moves things from the realm of unfeasible to definitely workable.