Brian Grant

Others, such as for load balancing, built independent services with their own service APIs and configuration mechanisms. This enabled teams to evolve their services independently, but created a heterogeneous, inconsistent management surface.

Omega supported an extensible object model, and @davidopp had proposed putting an API in front of the persistent store, as we later did in Kubernetes, but it wasn't declarative. Separate work on a common configuration store was discontinued as Google Cloud became the focus

GCP was comprised of independent services, with some common standards, such as the org hierarchy and authz. They used REST APIs, as the rest of the industry, and gRPC didn't exist yet. But, GCP’s APIs were not natively declarative, and Terraform didn’t exist, either

@jbeda proposed layering an aggregated config store/service with consistent, declarative CRUD REST APIs over underlying GCP and third-party service APIs. This sort of later evolved into Deployment Manager.

This is one of the tricky parts of an eventually consistent declarative system. There is no strong idea of "done". When you ask for a pod to be created the *resource* is created in the data store. But the actions implied by that resource aren't complete.

But there is no well understood idea of what "complete" means for a resource. There *are* no terminal states. We can say that a resource has converged and that the controller is idle but that could change at any time.

In addition the resource may *never* converge. That could be the result of a temporary condition (that resolves in 15 seconds) or the result of a permanent condition. Sometimes the controller/system in question can't tell.