"tl;dr: Multiprocessing in Python is crippled by pickle's poor function serialization." @mrocklin http://matthewrocklin.com/blog/work/2013/12/05/Parallelism-and-Serialization/ …
-
-
@posco@tdhopper@mrocklin I haven't looked deeply at it but it seems like Julia has a decent story there. /cc@StefanKarpinski -
@avibryant@posco@tdhopper@mrocklin this is true – Julia's closure serialization "just works" -
@StefanKarpinski@posco@tdhopper@mrocklin how do you avoid capturing the whole world? -
@avibryant@posco@tdhopper@mrocklin local state is capture but global state isn't; you have to arrange for global state to match -
@StefanKarpinski@posco@tdhopper@mrocklin global state meaning the top level? Any thoughts of delimiting it more explicitly? -
@avibryant@StefanKarpinski@posco@tdhopper@mrocklin why, delimited dynamic binding, of course ;-) http://okmij.org/ftp/Computation/dynamic-binding.html#DDBinding … -
@msimoni@StefanKarpinski@posco@tdhopper@mrocklin well, this is about closures, not continuations. What I want is delimited lexical scope -
@avibryant@StefanKarpinski@posco@tdhopper@mrocklin I suspect there's no good solution; Kali had to do dist. GC http://www.cs.purdue.edu/homes/suresh/papers/toplas95.ps.gz … - 2 more replies
New conversation -
-
-
@avibryant@posco@tdhopper@mrocklin c.f. Cloud Haskell -
@marius@avibryant@posco@tdhopper@mrocklin -- Cloud Haskell is shared nothing, good for most things, but you don't need Haskell for that. -
@coreload@avibryant@posco@tdhopper@mrocklin we’re talking about closure serialization; not about distribution models. -
@marius@avibryant@posco@tdhopper@mrocklin -- if you're not worried about synchronization, copying a closure environment is trivial. -
@avibryant Can't get it to stick in my brain, is problem capturing too much that you don't need? Or that you actually need too much? -
@peterseibel mostly the former. But also having good ways to need less (lazy init helps a lot). -
@avibryant Closing over in scope vars that aren't actually used? Or over things that are used but which also lead to big things that aren't? - 2 more replies
New conversation -
-
-
@avibryant@posco@tdhopper@mrocklin You've seen Scala spores, right? https://speakerdeck.com/heathermiller/spores-distributable-functions-in-scala … -
@noelwelsh@avibryant@tdhopper@mrocklin yes, but I still haven't understood why this is needed. Why not default on val capture? -
@posco@avibryant@tdhopper@mrocklin I don't know but I can make up an answer: to control mem use. Cc@heathercmiller -
@noelwelsh@posco@avibryant@tdhopper 3) control whats allowed to be captured w/in delim lex. scope 4) in a staticly-verifiable way w types -
@heathercmiller@noelwelsh@avibryant@tdhopper these things are wins, I just wish the compiler made it less necessary on immutable captures -
@posco@noelwelsh@avibryant@tdhopper what specifically do you mean by "immutable capture"? -
@heathercmiller@noelwelsh@avibryant@tdhopper I mean when you are just capturing this.someVal. Don't capture this, just get the val. -
@posco@noelwelsh@avibryant@tdhopper it's not clear whether you could detect the absence of lazyness and then special-case that.
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.