Brian Grant

Kubernetes Borg/Omega history topic 8: Declarative configuration and Apply. Inside Google, the most used configuration approach for Borg is the Turing-complete Borg Configuration Language (BCL). You can see a snippet of BCL on slide 7 in this deck: http://www.inf.ed.ac.uk/teaching/courses/exc/slides/Wilkes.pdf …

Millions of lines of BCL have been written. A fair amount of BCL was devoted to configuring application command-line flags, which was the most common way to figure server binaries, which is crazy IMO, but the practice sadly carried over to Kubernetes components

BCL was evaluated and instantiated using the borgcfg CLI, which supports commands like up, down, and update. Logic to diff and merge, perform rolling updates, and otherwise update the live state was embedded in the tool. Logic for common generation functions was written in BCL

This created a monolithic configuration and tool ecosystem. Even frameworks like mapreduce and services on top of Borg like BorgCron had to use BCL and borgcfg to interact with Borg. Getting-started tools generated BCL

A Python-based language was later developed, also. It interfaced with the update logic via a protobuf that wasn't quite the same as Borgmaster's. Other languages, such as Ruby, weren't used in Google. Several new Borg config languages were developed, but none were approved

Borgcfg didn't provide configuration packages. Shared templates were unversioned and directly imported from their homes in the monorepo, which inflicted churn on their consumers. There were also no "stacks" or lifecycle directives, so a number of imperative updates were needed

In Kubernetes, we wanted to decouple configuration authoring and generation from updates to the desired state via the API, so that users could express configuration using languages and tools familiar to them: Jinja, Python, Ruby, Javascript, Terraform, Ansible, whatever