Brian Grant

Even though Borg tasks are designed to be resilient, this could get pretty disruptive. Rate-limiting maintenance tasks independently isn't efficient if you have dozens of them, and it's not always feasible to perform all types of maintenance at the same time.

Even if rate-limiting machine disruptions, it's also possible for the same task to get disrupted over and over again, like a can being kicked down the road. Hence, the Safe Removal Service (aka SRSly -- pronounced seriously) was developed by SRE. (SRE builds lots of automation.)

SRSly kept track of how often tasks of the same Borg Job were disrupted (aka evicted). Maintenance automation queried SRSly regarding all tasks scheduled on a machine before taking it out of service. This enabled Borg to provide a SLO on task disruption.

Borgmaster, however, did not know about SRSly. Instead, all critical/production workloads were changed to run with the same priority so that they wouldn't preempt each other. Doing that for every Borg Job in the company was extremely painful -- more on priority/preemption later

For Omega, we developed a model that could be applied both during task preemption to run a higher-priority task and eviction for maintenance -- disruption counters. There was a time dimension that ended up not being effective due to constant changes, so we dropped it in K8s