Brian Grant

Compressible resources like CPU can be made available quickly by the kernel with low impact to the threads that were interrupted, provided it knows which threads urgently need the resources and which ones don't. We call this latency sensitive and latency tolerant respectively

In order to reallocate incompressible resources quickly, threads need to be killed, which is obviously not low impact. (For memory, in Linux this is done by the OOM killer.) This was why Borg used priority (production priority vs not) to make memory oversubscription decisions

Borg's resource reclamation approach is described by the paper: reservations based on observed usage were computed and oversubscribed resources (latency-tolerant cpu and non-production memory) were tallied against reservations whereas guaranteed ones used limits. Complicated.

Vertical autoscaling (VA) added even more complexity. VA changed limits, but left its own padding to provide slack for reaction time and observation of demand. Ad hoc mechanisms were added to disable limit enforcement for each resource, creating a notion similar to request in K8s

Request==Limit implies guaranteed resources (not oversubscribed). Request<Limit implies burstable (oversubscribed). Zero request implies best effort. Borg scheduled best effort using reservation, but no throughput guarantees could be made in practice

This is described in the resource model design (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/resources.md …) and the QoS proposal (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/resource-qos.md …), including the mapping to OOM scores. The mapping to cgroup cpu shares is described in the pod resource design (https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/pod-resource-management.md …).