Brian Grant

Like most internal Google services, Borgmaster had an imperative, unversioned, monolithic RPC API built using the precursor to http://grpc.io , Stubby. It exposed an ad hoc collection of operations, like CreateJob, LookupPackage, StartAllocUpdate, and SetMachineAttributes

Hundreds to thousands of clients interfaced with this API. Many of them were asynchronous controllers or monitoring agents, as discussed in previous threads, and there was a simple command-line tool, and two widely used configuration CLIs

The APIs were manually mapped into the two Turing-complete configuration languages, and there was also a hand-crafted diff library for comparing the previous and new desired states. The sets of concepts, RPC operations, and configurable resource types were not easily extended

Some extensions of the core functionality, such as for batch scheduling and vertical autoscaling, used the Borgmaster as a configuration store by manually adding substructures stored with Job objects, which were then retrieved by polling Jobs.

Others, such as for load balancing, built independent services with their own service APIs and configuration mechanisms. This enabled teams to evolve their services independently, but created a heterogeneous, inconsistent management surface.

Omega supported an extensible object model, and @davidopp had proposed putting an API in front of the persistent store, as we later did in Kubernetes, but it wasn't declarative. Separate work on a common configuration store was discontinued as Google Cloud became the focus

GCP was comprised of independent services, with some common standards, such as the org hierarchy and authz. They used REST APIs, as the rest of the industry, and gRPC didn't exist yet. But, GCP’s APIs were not natively declarative, and Terraform didn’t exist, either

GCE was RESTful and, to some degree, was declarative. We didn't model all transitions as declarative though. We also didn't have a clear separation of the parameters that are set by the users and those that are status or set by the system.