Very fascinating result here. Doing a "for line in sys.stdin" in Python is much slower than sys.stdin.buffer.read and I'm not sure why. The I/O bottleneck is completely removed when sending chunks of ndjson to multiprocessing workers instead of reading line by line.
Conversation
Replying to
For instance ...
import sys
buf = b''
while True:
data = sys.stdin.buffer.read(4096*64)
l = len(data)
pos = data.rfind(b'\n')
chunk = buf + data[0:pos]
buf = data[pos:]
if not data:
break
3
This will send large chunks of ndjson (none broken in the middle of a json object) and the chunk can be sent to a multiprocessing worker thread and then split and decoded there. Should be able to process hundreds of thousands a second if the I/O is fast enough. We'll see.