it actually can work slower if you use too many cores. Try with 2 or 4 to see if it's speeding up.
-
-
-
I have 32 on my machine tho..?
-
I tried with 2 and 4 also just to be clear.
-
I've just seen your code. Not sure if it might be due to apply_async? I usually do this: https://pastebin.com/uJVvpYqv
-
I'll try your way. But like
@seaandsailor was saying it could be that I'm not doing enough on each worker? Turbo thanks for code I'll try.
-
OK, so I don't actually understand your way, sorry. You have to bear in mind I used multiprocessing for the first time yesterday.

-
for example where you define your arguments why are you using itertools.product?
-
does that mean I have to use itertools.combinations? my loop is outside all this... hmm...
- 2 more replies
New conversation -
-
-
shared memory, scheduling and inter-thread communication (applies to all languages)
-
may need to modify data structures or algorithms to mitigate
-
I know but like I have it working and it's memory safe unlike scipy/sklearn versions BUT NO IDEA how to get to the data.
-
get the data from where?
-
OK, hang on — many have tried to help and I got this far mostly without ever using this library.
-
Actually I might not be able to share right now because of how busy I am. I am trying to parallelise something & failing abysmally here are
-
some of my attempts — currenytly trying something new as I'm 100% stuck there:https://github.com/oliviaguest/pairwise_distance/blob/master/pairwise_distance.py …
-
my new tactic is also failing so if you can see the light for any of these attempts that wd be awesome — going to keep trying a little more
- 2 more replies
New conversation -
-
-
How much work are you doing in each process and how much information are you passing between processes?
-
I am trying to do a simple distance metric for each two points in a huge array and I think I am passing nothing.
-
Shall I share my code? Would that help?
-
If it it would, here it is! I am a total noob with this in Python so I'm probably doing something terribly wrong!https://github.com/oliviaguest/pairwise_distance/blob/master/pairwise_distance.py …
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.
