Maybe wrong but I don't think what I was doing can be done using dask wi/out 1st writing to disk actually... @seaandsailor @usethespacebar
So far I have tried the latter and it's super slow. I will try writing the pairs of items to disk though and doing delayed loading.
-
-
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
don't forget you're simulating data but in real life it would be already on disk. Dask is meant to be used when you cannot load all the data
-
actually I figured out how you do it w/out storing on disk — create np arrays in a loop & concatenate http://dask.pydata.org/en/latest/array-stack.html#concatenate … cc
@neuromusic -
but technically I would need to have the same method for on disk as I haven't got the pairs stored on disk either only single points

-
so iterating over combs might be useful here — but this would be in contrast to the method currently being used with multiprocessing

-
Let's see!
End of conversation
New conversation -
-
-
Ahh gotcha
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.