The problem isn’t MongoDB per se (I’m using python to collect)—it’s that I don’t have a queuing solution in place and the script drops tweets or fails when tweets come in too quickly. I looked at RabbitMQ but hadn’t managed to get it working.https://twitter.com/faineg/status/1203111652956217349 …
-
-
Thanks :-) twarc isnt the prettiest code ever, but it does have a pretty simple CLI, and can be easily used as a library. I guess the best thing is that it has logic to recover from dropped connections.
-
*thinks back to that time I was writing twarc in go without knowing about twarc* (Although, my golang lib has really progressed for my backend — almost no allocations. Just no need to for a tool that does the same thing, especially when the existing tool has good maintainers.)
- 3 more replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.