A long time ago, I worked for a Japanese company which had a Alert Lots Of People function available to it. We drilled on its use. Since it's instructive (this thread may be longish):
-
-
B: "Confirming you are attempting to log into [production / test environment]." A: "Login complete." B: "I verify you are logged into production. Proceed." A: "I am queueing up an alert, entitled
$TEXT." B: "Confirm to me that your expected text is [reads screen]." A: "Confirm"Show this thread -
They then verify that the alert was in fact sent and verify it was received on one of the canaries, using a similar level of pedanticness.
Show this thread -
This could be circumvented by any engineer, on their own authority, if there was a danger to human life. (e.g. "I became aware of a tsunami at 3 AM in the morning and could not raise a second within 15 seconds.") We thought about this process *a lot*, and drilled it.
Show this thread -
You can file this under "an anecdote from an engineering culture which does a few things really, really well"; this is one of them.
Show this thread -
"Was it an actual, physical book?" Well, technically, a binder, in case we ever needed to update procedures. "Why not have it on an electronic system?" Because binders have 100% uptime even if office/remote servers/etc are presently rattling in the wake of e.g. earthquake.
Show this thread -
"What happens if the Internet connection was down between you and the system?" This is a question that gets perilously close to that I Can't Tell You territory, but web developers who have ever coded "Fire X if we don't get a heartbeat from Y" can probably guess.
Show this thread -
The above is offered in the spirit of improving plans at your current/future workplaces and not as even an implied criticism of any other systems. When it really counts, there are only two teams: humanity and disaster. We’re all in it together on Team Humanity.
Show this thread
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.