Conversation

After nearly a decade of frustration with Python's multiprocessing library, I recently tried

and it's fantastic! I am using essentially none of its features (just a parallel map over some parameter ranges + shared read-only input buffers), but it's so much nicer.

Nicholas Schiefer

@nschiefer

If, like me, you find yourself reaching for multiprocessing and then feel frustrated, check it out!

ray.io

Productionizing and scaling Python ML workloads simply | Ray

Scale your compute-intensive Python workloads. From reinforcement learning to large-scale model serving, Ray makes the power of distributed compute easy and accessible to every engineer.

4:50 PM · Jul 30, 2021Twitter Web App

Replying to

I like it too but failed to find patterns/examples to do a multi-stage map eg pipeline w queues. Maybe that's what the new Dataset api addresses...

Replying to

My use case was super simple so I’m not really sure. I think the Dataset API seems like a good fit for that, though!

Show more replies