Kubernetes Borg/Omega history topic 4: Workload controllers. Before I get into the history, some conceptual background may be useful, since the underpinnings come up in many Cloud Native contexts. The key is to explicitly model state such that it can be externally manipulated.
-
-
In http://issues.k8s.io/1518 , we started to discuss what became DaemonSet. The key decision was whether to add more functionality to ReplicationController or to create new resource types. Users of other systems were concerned about the complexity of using multiple types
Show this thread -
Borg had supported just one workload "controller", the Job. (I'll address the differences between Borg's synchronous state machine and the Kubernetes async controllers later.) It's described well by the Borg paper:https://ai.google/research/pubs/pub43438 …
Show this thread -
Job, an array of Tasks, is used for elastic services, agents that ran on every node, batch workloads, and stateful workloads. Consequently, it has a large number of settings, and additional, external controllers are needed in order to support these different workloads.
Show this thread -
For instance, for the daemon use case, a special controller / autoscaler is needed to ensure that the Job has a sufficient number of Tasks to cover all the machines, and cases where machines are removed from the middle of the array require special handling.
Show this thread -
And not only is Job is the first-class primitive rather than Tasks, but each Task has a stable identity, as with StatefulSet in Kubernetes. That overly constraining not just for daemons, but also for autoscaled workloads, CI workloads, graceful termination, debugging, etc.
Show this thread -
Job also includes published BNS records for tasks, which is the rough equivalent of Endpoints in Kubernetes. BNS records are stored in Chubby, where they can be watched. (I'll cover watch in K8s more generally later.)
Show this thread -
The decoupling of Pod, workload controllers, and Endpoints, and a precedent for multiple workload controllers in Kubernetes has proven very flexible, for supporting many, many types of workloads. There are now many application-specific workload controllers (aka Operators).
Show this thread -
Explicitly representing the PodTemplate as a separate object, as proposed in http://issues.k8s.io/170 , may also have been useful for these third-party controllers, but in practice the lack of support for that hasn't been a huge obstacle. (Well, the API exists, but is unused.)
Show this thread -
I proposed the idea of modeling workload controllers as loosely coupled sets of instances grouped using a label selector in June 2013, based on an 11-page analysis of Borg Job use cases, around the same timeframe as the original labels proposal.
Show this thread -
That partly inspired http://replicapool.googleapis.com also, though the lack of labels in GCE at the time made implementing the full model infeasible.
Show this thread -
Aside on "template": a "template" is a pattern used to make copies of the same shape. I think the Kubernetes "Pod template" usage is true to that colloquial definition, but typical CS usage implies parameterization and/or macro expansion, so maybe "prototype" would be better
Show this thread -
The idea of explicitly modeling state so that it can be externally controlled and observed is a key principle of Cloud Native. I originally included it in a longer form of the definition I wrote for CNCF: https://github.com/cncf/toc/blob/master/DEFINITION.md …
Show this thread -
The principle can also be applied to workflow systems and configuration management (e.g., see https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/declarative-application-management.md …). Embodying these as code is powerful, but with great power comes great responsibility, since it obstructs external tooling and automation
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.