Kubernetes Borg/Omega history topic 5: Asynchronous controllers. Borgmaster had synchronous, transactional, edge-triggered state machines. We had challenges scaling, evolving, and extending them.
Show this thread
-
As a result, when new teams worked on new functionality, such as batch scheduling and autoscaling, they built it into external components, which were asynchronous. Ingestion of status from nodes (Borglets) was asynchronous, as well. Omega embraced asynchronous controllers
Show this thread
Omega represented desired state and observed state in separate records in its transactional Paxos store. This made it harder to assemble a picture of what was going on. In Kubernetes, we decided to represent status in the same object as spec, in v1beta3:http://issues.k8s.io/1225
-
-
We also fully embraced the controller model, even for Kubelet, by making Kubelets report back to apiserver (http://issues.k8s.io/156 ) and patch status (http://issues.k8s.io/2726 ) so that the API could be used as the source of truth by other controllers.
Show this thread -
Rather than rigid fine-grained state enumerations that couldn't be evolved, we initially adopted simple basic states that could report open-ended reasons for being in each state (http://issues.k8s.io/1146 ), and later non-orthogonal, extensible conditions (http://issues.k8s.io/7856 ).
Show this thread -
The entire system can now be described as an unbounded number of independent asynchronous control loops reading and writing from/to a schematized resource store as the source of truth. This model has proven to be very resilient, evolvable, and extensible.
Show this thread
End of conversation
New conversation -
-
-
The spec vs status thing was always there from the start. It was just called desiredstate vs currentstate in early versions of the API. I never intended these to have the same schema. This was a big lesson learned from the GCE API.
-
The early specs for the API made this clear but these docs were before things were open and aren't public. If I were still at Google I'd dig them up.
End of conversation
New conversation -
-
-
It's represented as the same object, but Status is a subresource (with its own storage in etcd) is it not?
-
It is a subresource, to prevent spec and status changes from accidentally stepping on each other and to enable distinct ACLs, but they don't have separate storage yet. Updating 2 resources in etcd2 (e.g. BoundPod) was painful.http://issues.k8s.io/8625
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.