I'm experimenting with automatic lineage tracking in R: https://github.com/jwills/lineage Thinking in monads actually helps a lot. #shock #rstats
@posco @josh_wills @jaykreps and Kinesis is basically AWS Samza. But Spark Streaming is also a planner, no?
-
-
@avibryant@posco@josh_wills I think Kinesis is more similar to Kafka (partitioned log storage) rather than any processing layer... -
@jaykreps@posco@josh_wills if you look at the docs for their java client libs, it's very like Samza, I think. http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-record-processor-app.html …
End of conversation
New conversation -
-
-
@avibryant@josh_wills@jaykreps planner, yes. --@matei_zaharia how much plan optimizing does spark streaming do? -
@posco@avibryant@josh_wills@jaykreps Similar to Spark; it does pipelining, partitioning, incremental operators, data locality
End of conversation
New conversation -
-
-
@posco@josh_wills@jaykreps also, check out how Samza does state; there's some interesting stuff there that goes beyond "streaming MR".Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.