Kubernetes Borg/Omega history topic 14: Computational Quality of Service (QoS) and oversubscription. What are they, why would you want them, and how is QoS different than priority? On the last point, it's distinguishing importance and urgency.
-
-
Borg's resource reclamation approach is described by the paper: reservations based on observed usage were computed and oversubscribed resources (latency-tolerant cpu and non-production memory) were tallied against reservations whereas guaranteed ones used limits. Complicated.
Show this thread -
Vertical autoscaling (VA) added even more complexity. VA changed limits, but left its own padding to provide slack for reaction time and observation of demand. Ad hoc mechanisms were added to disable limit enforcement for each resource, creating a notion similar to request in K8s
Show this thread -
In K8s, I wanted something simpler, to directly convey the desire for oversubscription and bursting flexibility. The discussion started way back in http://issues.k8s.io/147 and http://issues.k8s.io/168 . The model we settled on was determined by looking at limits and requests
Show this thread -
Request==Limit implies guaranteed resources (not oversubscribed). Request<Limit implies burstable (oversubscribed). Zero request implies best effort. Borg scheduled best effort using reservation, but no throughput guarantees could be made in practice
Show this thread -
This is described in the resource model design (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/resources.md …) and the QoS proposal (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/resource-qos.md …), including the mapping to OOM scores. The mapping to cgroup cpu shares is described in the pod resource design (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/pod-resource-management.md …).
Show this thread -
Some work on Vertical Pod Autoscaling for Kubernetes has started: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/autoscaling/vertical-pod-autoscaler.md …. There have been proposals to implement oversubscription also (https://github.com/kubernetes/enhancements/issues/355 …). As for horizontal scaling, resource monitoring infrastructure is a prerequisite
Show this thread -
If managing cluster-level sharing using ResourceQuota and LimitRange, oversubscription can be done at that level also. The original designs were described by https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/admission_control_limit_range.md … and https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/admission_control_resource_quota.md …, with improvements in https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-management/resource-quota-scoping.md …
Show this thread -
Ok, this topic doesn't fit into a Twitter form factor very well. Maybe some day I'll get around to writing this up more in long form. For now, that's about all I have time for, but questions are welcome
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.