Kubernetes Borg/Omega history topic 5: Asynchronous controllers. Borgmaster had synchronous, transactional, edge-triggered state machines. We had challenges scaling, evolving, and extending them.
Show this thread
High-cardinality resource instances could exceed what could be done in a single transaction. Addition of new states broke clients. Unobserved changes could cause unexpected state transitions. Adding new resource types was hard, and would have had to be added to monolithic files.
-
-
As a result, when new teams worked on new functionality, such as batch scheduling and autoscaling, they built it into external components, which were asynchronous. Ingestion of status from nodes (Borglets) was asynchronous, as well. Omega embraced asynchronous controllers
Show this thread -
Omega represented desired state and observed state in separate records in its transactional Paxos store. This made it harder to assemble a picture of what was going on. In Kubernetes, we decided to represent status in the same object as spec, in v1beta3:http://issues.k8s.io/1225
Show this thread -
We also fully embraced the controller model, even for Kubelet, by making Kubelets report back to apiserver (http://issues.k8s.io/156 ) and patch status (http://issues.k8s.io/2726 ) so that the API could be used as the source of truth by other controllers.
Show this thread -
Rather than rigid fine-grained state enumerations that couldn't be evolved, we initially adopted simple basic states that could report open-ended reasons for being in each state (http://issues.k8s.io/1146 ), and later non-orthogonal, extensible conditions (http://issues.k8s.io/7856 ).
Show this thread -
The entire system can now be described as an unbounded number of independent asynchronous control loops reading and writing from/to a schematized resource store as the source of truth. This model has proven to be very resilient, evolvable, and extensible.
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.