could you jam them together into one big temp file with eg the cat command in the Shell before u run python?
is there an efficient way, in python, to read many (10^5) small files to memory? using sequential open() -> major IO bottleneck, even w/ SSD
-
-
-
hmmmm. YES! I'll try this later. clever, thank you. times like this I wish I'd had a formal CS education

-
is just technique, you learn them as you go.
-
yeah formal CS education is mostly about asymptotic algorithm performance lol
-
eerily similar to classical, generally useless econometric theory coursework
-
i mean, damn it, if i suffered through the proof of complexity of tarjans disjoint union find algorithm
-
it had to be worth something, right?
-
( iirc it scales with the inverse of Ackerman's function)
- 2 more replies
New conversation -
-
-
threads
-
guh, I'd dumb. 'sequential' inaccurate. have 24 parallel processes; each calls open, performs task on binary input, repeat.
-
jc, *I'm*. anyway hope was that there might be a way to mass-access drive data rather than repeated, maybe-colliding reads.
-
cool. In that case I probably got nothing, except, you should be able to beat 24 w thread parallelism
End of conversation
New conversation -
-
-
PUSH THE BUTTON AND GO FEED THE CAT
-
I should print this and frame this and my cat thanks youpic.twitter.com/9i4xb067wa
-
I ONCE DID AN HAPHAZARD ART INSTALLATION OF THIS
-
MEANING I TOOK SEVERAL POORLY FOCUSED PHOTOS OF MYSELF PRETENDING TO BE A CAT LICKING ITSELF
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.