The problem isn’t MongoDB per se (I’m using python to collect)—it’s that I don’t have a queuing solution in place and the script drops tweets or fails when tweets come in too quickly. I looked at RabbitMQ but hadn’t managed to get it working.https://twitter.com/faineg/status/1203111652956217349 …
-
Show this thread
-
Replying to @mountainherder
DM me later friends, I'd love to help! I have,,, ,,,a lot of collection pipelines.
2 replies 0 retweets 2 likes -
Replying to @generativist @mountainherder
Although, the best way is probably just to use twarc (h/t
@edsu) which is designed for this type of use case. Save it to jsonl then process it later. $ twarc filter demDronesDoe > stream.jsonlhttps://github.com/DocNow/twarc1 reply 0 retweets 1 like -
Replying to @generativist @mountainherder
Thanks :-) twarc isnt the prettiest code ever, but it does have a pretty simple CLI, and can be easily used as a library. I guess the best thing is that it has logic to recover from dropped connections.
1 reply 0 retweets 2 likes -
Replying to @edsu @mountainherder
*thinks back to that time I was writing twarc in go without knowing about twarc* (Although, my golang lib has really progressed for my backend — almost no allocations. Just no need to for a tool that does the same thing, especially when the existing tool has good maintainers.)
1 reply 0 retweets 1 like -
Replying to @generativist @mountainherder
Ed Summers Retweeted Bergis Jules 🇱🇨
Thanks for your support John! I think there could still be a sweet spot for a Go client that allowed multiple different connections with different credentials. Btw, I dont know if you saw this, but any short message of support would be appreciated!https://twitter.com/BergisJules/status/1201901571937718272?s=20 …
Ed Summers added,
Bergis Jules 🇱🇨 @BergisJulesWe would love to hear from you for our grant report if you’ve used@documentnow’s twarc, DocNow demo, Catalog, Hydrator. Or maybe our work has inspired yours, you attended our events/workshops, or you learned something in our Slack channel. Send us a testimonial at info@docnow.io1 reply 0 retweets 1 like -
Replying to @edsu @mountainherder
I'm running out to
@indiewebcamp right now but I'll absolutely do this. (If you don't hear from me, I forgot and a nudge woudn't be inappropriate.)2 replies 0 retweets 2 likes
Oh also, let me think about that! I can rip out the core and turn it back into an open library. That support is largely built in because my use-case is writing twitter apps as services.
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.