A bunch of people have been asking me about Spotify moving away from Luigi, the open source workflow scheduler and I built in 2011-2015: engineering.atspotify.com/2022/03/why-we – will put some thoughts in a thread
Conversation
1. Luigi afaik was the first workflow scheduler to gain any sort of mainstream adoption. It was clear to me it solved a set of problems that many people had. Over the years it's grown to 15k stars on Github.
1
8
2. The design decision that I was the happiest about is probably how Luigi it super functional in nature. Everything is just functions (defined in an awkward way using classes) depending on other functions.
1
8
3. The by far biggest missing feature is that Luigi in itself doesn't trigger jobs – it relies on something else to do. The use case I had (Spotify's music recs) was somewhat atypical in retrospect – one super-complex DAG with little/no external dependencies.
1
6
4. I could be wrong but my feeling is that the main reason people prefer Airflow is that it focused on exactly that. It makes it easy to build lots of of simple DAGs that each get triggered independently.
Quote Tweet
Now that I have an iPad with a pencil again, I’m going to take a break from work by creating a series of poorly drawn but honestly named machine learning tool logos.
Show this thread
2
13
5. I actually talked to some VCs about turning Luigi into a startup back in 2015. The reason I decided against it is it seems quite hard to monetize workflow scheduling. It's hard to charge a lot of money for just a "control plane in the cloud". I could be wrong though!
4
1
11
6. That being said, it's clear from Spotify that workflow scheduling is extremely sticky. Moving away from one tool to another took Spotify many many years (I left Spotify in 2015 and stopped maintaining Luigi shortly after). So retention can be extremely high.
2
1
10
yeah its right on top of the “if it aint broke dont fix it” stack. but glad we figured out the monetization piece.
1
1