Brian Grant on Twitter: "And the Safe Removal Service was also mentioned in Google's "VM Live Migration at Scale" paper in VEE 2018: https://t.co/GOd7VOBhNi"

Country	Code	For customers of
United States	40404	(any)
Canada	21212	(any)
United Kingdom	86444	Vodafone, Orange, 3, O2
Brazil	40404	Nextel, TIM
Haiti	40404	Digicel, Voila
Ireland	51210	Vodafone, O2
India	53000	Bharti Airtel, Videocon, Reliance
Indonesia	89887	AXIS, 3, Telkomsel, Indosat, XL Axiata
Italy	4880804	Wind
3424486444	Vodafone
» See SMS short codes for other countries

Brian Grant‏ @bgrant0607 31 May 2019

Report Tweet
Report NetzDG Violation

SRSly kept track of how often tasks of the same Borg Job were disrupted (aka evicted). Maintenance automation queried SRSly regarding all tasks scheduled on a machine before taking it out of service. This enabled Borg to provide a SLO on task disruption.

1 reply 0 retweets 6 likes

Show this thread

Brian Grant‏ @bgrant0607 31 May 2019

Report Tweet
Report NetzDG Violation

Borgmaster, however, did not know about SRSly. Instead, all critical/production workloads were changed to run with the same priority so that they wouldn't preempt each other. Doing that for every Borg Job in the company was extremely painful -- more on priority/preemption later

1 reply 0 retweets 6 likes

Show this thread

Brian Grant‏ @bgrant0607 31 May 2019

Report Tweet
Report NetzDG Violation

For Omega, we developed a model that could be applied both during task preemption to run a higher-priority task and eviction for maintenance -- disruption counters. There was a time dimension that ended up not being effective due to constant changes, so we dropped it in K8s

1 reply 1 retweet 6 likes

Show this thread

Brian Grant‏ @bgrant0607 31 May 2019

Report Tweet
Report NetzDG Violation

I think I first mentioned this in Kubernetes in my big scheduling braindump comment: https://github.com/kubernetes/kubernetes/issues/4301#issuecomment-74355529 …. It came up again when I proposed maxUnavailable to moderate concurrent disruptions caused by updates during the design of Deployment:https://github.com/kubernetes/kubernetes/pull/12236#discussion_r36501373 …

1 reply 0 retweets 5 likes

Show this thread

Brian Grant‏ @bgrant0607 31 May 2019

Report Tweet
Report NetzDG Violation

That discussion was forked into https://github.com/kubernetes/kubernetes/issues/12611 …. Around that time, Matt Liggett (https://github.com/kubernetes/kubernetes/pulls?q=is%3Apr+author%3Amml+is%3Aclosed …) joined the GKE team from Borg SRE (woo hoo!). One of the first things Matt worked on was improving node drains:https://github.com/kubernetes/kubernetes/issues/6080 …

1 reply 0 retweets 7 likes

Show this thread

Brian Grant‏ @bgrant0607 31 May 2019

Report Tweet
Report NetzDG Violation

Together with @davidopp and @erictune4, we folded disruption budgets into the rescheduling design proposal: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/rescheduling.md#disruption-budget …. (Rescheduling deserves its own thread -- I'll do that one next.) Implementation began in https://github.com/kubernetes/kubernetes/pull/24697 … andhttps://github.com/kubernetes/kubernetes/pull/25551 …

1 reply 0 retweets 3 likes

Show this thread

Brian Grant‏ @bgrant0607 31 May 2019

Report Tweet
Report NetzDG Violation

PodDisruptionBudget is now documented: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/ … and https://kubernetes.io/docs/tasks/run-application/configure-pdb/ …. Try it out and give us feedback on how well it works for you. We're looking to advance it from beta to GA:https://github.com/kubernetes/enhancements/issues/85 …

3 replies 2 retweets 11 likes

Show this thread

Brian Grant‏ @bgrant0607 31 May 2019

Report Tweet
Report NetzDG Violation

You can safely drain a node with kubetctl drain: https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/ …. Node upgrades and the cluster autoscaler in Google Kubernetes Engine (GKE) also respect PodDisruptionBudget. The latter is documented here:https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler …

1 reply 1 retweet 5 likes

Show this thread

Brian Grant‏ @bgrant0607 31 May 2019

Report Tweet
Report NetzDG Violation

Node upgrade behavior is documented here:https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades …

1 reply 1 retweet 4 likes

Show this thread

Brian Grant‏ @bgrant0607 31 May 2019

Report Tweet
Report NetzDG Violation

And more about the Google SRE philosophy behind automation can be found in the SRE book: https://landing.google.com/sre/sre-book/chapters/automation-at-google/ …

1 reply 1 retweet 7 likes

Show this thread

Brian Grant‏ @bgrant0607 31 May 2019

Report Tweet
Report NetzDG Violation

And the Safe Removal Service was also mentioned in Google's "VM Live Migration at Scale" paper in VEE 2018: https://dl.acm.org/citation.cfm?id=3186415 …

12:31 PM - 31 May 2019

1 Retweet
4 Likes

2 replies 1 retweet 4 likes

Loading seems to be taking a while.

Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.

Tweets

Brian Grant

@bgrant0607

Tweets

Loading seems to be taking a while.

false

Saved searches

Tweets

Brian Grant

@bgrant0607

Tweets

Go to a person's profile

Saved searches

Promote this Tweet

Block

Tweet with a location

Your lists

Create a new list

Copy link to Tweet

Embed this Tweet

Embed this Video

Preview

Why you're seeing this ad

Log in to Twitter

Sign up for Twitter

Not on Twitter? Sign up, tune into the things you care about, and get updates as they happen.

Two-way (sending and receiving) short codes:

Confirmation

Welcome home!

Tweets not working for you?

Say a lot with a little

Spread the word

Join the conversation

Learn the latest

Get more of what you love

Find what's happening

Never miss a Moment

Loading seems to be taking a while.

Promoted Tweet

false