Brian Grant

Kubernetes Borg/Omega history topic 6: Watch. This is a deep topic. It's a follow-up to the controller topic. I realized that I forgot to link to the doc about Kubernetes controllers: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md …

One of the first things I did when joining the Borgmaster team back in 2009 was to parallelize the handling of read requests. Something like 99% of requests were reads, primarily from polling external controllers and monitoring systems.

Only BNS (analogous to K8s Endpoints) was written to Chubby, which enabled replicated caching and update notification. That enabled it to scale to much larger numbers of readers (~every container in Borg) and reduced latency, which for polling could be tens of seconds

Watch-like notification APIs (aka sync and tail) were common for storage systems such as Chubby, Colossus, and Bigtable. In 2013, a generalized Watch API was designed so that each system wouldn't need to reinvent the wheel. A variant "Observe" added per-entity sequencing

We built Kubernetes upon Etcd due to its similarities to Chubby and to the Omega store. When we exposed Etcd's watch (https://coreos.com/etcd/docs/latest/learning/api.html …) through the K8s API, we let more Etcd details bleed through than originally intended. We need to clean up some of those details soon

The Kubernetes model is described here: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture/resource-management.md#declarative-control …

Some other systems use message buses for notifications. Why didn't we? Controllers need to start from the initial state, we also don't want them to fall behind or operate on state that's too stale, and they need to be able to handle "missed" events -- the level-based rationale