I tried 32 because I have 32 cores, does that make sense and YES it is a lot faster but not faster than the other two.. hmmm...
-
-
Replying to @o_guest
I think the operation is so simple that you always see a slowdown due to the overhead of sending data to other processes and back.
1 reply 0 retweets 1 like -
Replying to @seaandsailor
the thing is though it will run out of memory the other way so this way has to be better for big data — right?
1 reply 0 retweets 1 like -
Replying to @o_guest
There's another issue. I'll try to fix it because it will be easier than explaining it on Twitter, then we can talk about it :)
1 reply 0 retweets 1 like -
-
Replying to @o_guest
Here's a faster version: https://gist.github.com/jfsantos/8184653991558e30a9eab8613a6ea20f …. The trick is to create a list with all combinations at once and use Pool.map.
1 reply 0 retweets 1 like -
Replying to @seaandsailor @o_guest
It's still not faster than the others, but not far from the loop. You'll see more performance gains when the function is more complicated.
2 replies 0 retweets 1 like -
Replying to @seaandsailor @o_guest
You can also improve runtime by using the scipy implementation and sending blocks of points to it instead of the entire X, then concat'ing
1 reply 0 retweets 1 like -
Replying to @seaandsailor
Oh — that sounds sensible — do you know how this is done?
2 replies 0 retweets 0 likes -
Replying to @o_guest
Several ways, but the easiest would be to use numpy.split (https://docs.scipy.org/doc/numpy/reference/generated/numpy.split.html …) to slice X, then iterate over the sliced matrix.
1 reply 0 retweets 1 like
I tried it that way and failed miserably. I guess I should think again. I think because I spent all day on this I should probably stop...
-
-
Replying to @o_guest @seaandsailor
I couldn't get my head around it and I guess I've exhausted myself now.
1 reply 0 retweets 0 likes - 8 more replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.
