Very fascinating result here. Doing a "for line in sys.stdin" in Python is much slower than sys.stdin.buffer.read and I'm not sure why. The I/O bottleneck is completely removed when sending chunks of ndjson to multiprocessing workers instead of reading line by line.
Conversation
For instance ...
import sys
buf = b''
while True:
data = sys.stdin.buffer.read(4096*64)
l = len(data)
pos = data.rfind(b'\n')
chunk = buf + data[0:pos]
buf = data[pos:]
if not data:
break
Replying to
This will send large chunks of ndjson (none broken in the middle of a json object) and the chunk can be sent to a multiprocessing worker thread and then split and decoded there. Should be able to process hundreds of thousands a second if the I/O is fast enough. We'll see.
Replying to
Assuming that reading stdin hits disk somewhere, in this approach you are reading 256K of data upfront from "slow media" and then finding the line in-memory in this 256K 'data' buffer. "for line in sys.stdin" is having to go to slow media for _each_ line. I assume your code ...
1
.. above is incomplete because I'd expect a for-loop after reading 256KB from stdin in which you keep looking for subsequent lines in 'data' and keep dispatching them to your pipelines. You stop only when what's left is a partial ndjson, then you "refill" from stdin. HTH.
1
Show replies