@posco @avibryant when designing Scalding pipes, did you consider inheriting the Cascading pipes? Why did you decide composing instead?
I've been wondering if a new Source abstraction should always return Execution[TypedPipe[T]]
-
-
the reasoning there is, roughly, that a Source is just a materialized job.
-
it's not clear that a downstream user should know/care about the "materialized" part.
-
the apis get fun fast. Path presence testing, toIterator =/= toIterableExecution, planner implications of push down map to src
-
yeah, definitely needs to be a lower level API for things like toIterator. But the default happy path should go through Execution.
End of conversation
New conversation -
-
-
yeah source could just be a TypedPipe factory. But I'm working on fixing FileSource not Source
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.