1. Email is the worst monitoring and alerting mechanism except for all the others. 2. Absence of a signal is itself a signal. 3. The severity of an incident is measured by the number of rules broken in resolving it.
-
-
Prikaži ovu nit
-
4. The mobile hotspot you're paying for so you can leave your house while you're oncall only works at home and in the office. 5. The only other person who knows how this works is also on vacation.
Prikaži ovu nit -
6. If a post-mortem follow-up task is not picked up within a week, it's unlikely to be completed at all. 7. That janky script you put together during the outage -- the one that uses expect(1) and 'ssh -t -t' -- now is the foundation of the entire team's toolchest.
Prikaži ovu nit -
8. NTP being off may not be a root cause, but it sure didn't help. 9. UTC or GTFO.
Prikaži ovu nit -
10. Your infrastructure uses a lot more self-signed certificates than you think. A lot more. In places that make you weep.
Prikaži ovu nit -
11. Self-signed certificates beget long lived certs, which beget lack of certificate validity monitoring, which begets curl -k, which begets a lack of certificate deployment automation, which begets self-signed certificates.
Prikaži ovu nit -
12. For any N applications, at most N/2+1 use the same certificate bundle. 13. The system you're troubleshooting doesn't use the one the tool you're troubleshooting it with does.
Prikaži ovu nit -
14. An API without a reference implementation and command-line client is called a gray box. 15. Restricted shells are not as restricted as you think. 16. Very few operations are truly idempotent.
Prikaži ovu nit -
17. "Asserting state" beats "monitoring for compliance" any day. 18. One in a Million is next Tuesday.https://docs.microsoft.com/en-us/archive/blogs/larryosterman/one-in-a-million-is-next-tuesday …
Prikaži ovu nit -
19. People give talks at conferences not to convince others that their work is awesome and totally worth the time and effort they put in, but themselves.
Prikaži ovu nit -
20. It's ok to use shell for complex stuff; it often times is easier, faster, and still less of a mess than juggling libraries and dependencies.
Prikaži ovu nit -
21. There's nothing wrong with Perl. 22. Ok, we all at times keep adding $, {, }, and @ in random places trying to make things work, but still.
Prikaži ovu nit -
23. Serverless isn't. 24. Y38K is already here, it's just not evenly distributed. 25. If you determine "human error" as the root cause, then you're doing it wrong.
Prikaži ovu nit -
26. Your network team has a way into the network that your security team doesn't know about. 27. And don't even as much as mention the serial console and IPMI networks, but boy are you glad you have 'em.
Prikaži ovu nit -
28. Blocking TCP port 53 traffic leads to very strange failures. Don't. 29. Somewhere in your infrastructure a service you didn't know uses DNS for endpoint discovery in a very surprising way. 30. Do
Not
Monkey
Around
With
/etc/hosts.Prikaži ovu nit -
31. If you break it, you own it - for now; if you fix it, you own it - forever. 32. Turning it off and on again is actually quite a reasonable way to fix many things. 33. A http://README.md in git is no substitute for a manual page that's shipped with your tool.
Prikaži ovu nit -
34. A search for a document you know exists will only turn up links to documents referencing but not actually linking to the one you're looking for.
Prikaži ovu nit -
35. The document you're looking for was marked as obsolete and not migrated to the new content management solution. 36. Sure, your current content management system sucks, but it's still better than the one you're moving to.
Prikaži ovu nit -
37. Nobody knows how git works; everybody simply rm -fr && git checkout's periodically.
Prikaži ovu nit -
38. There are very few network restrictions creative and determined use of ssh(1) port forwarding can't overcome. 39. This is both incredibly useful and concerning.
Prikaži ovu nit -
40. It is tempting to jump right into implementing a solution when the right thing may well be to not do the thing that requires the solution in the first place. 41. Turning things off permanently is surprisingly difficult.
Prikaži ovu nit -
42. "Ancient" is a very relative term when it comes to software and protocols. 43. "Obsolete" doesn't mean it's not in use and relied on.
Prikaži ovu nit -
44. The sets of systems online before and after a data center power outage only intersect. Some of the old systems coming online will immediately cause a different outage.
Prikaži ovu nit -
45. Some of your most critical services are kept alive by a handful of people whose job description does not mention those services at all.
Prikaži ovu nit -
46. After the initial "down for everybody or just me ermahgehrd Slack is down" drop, productivity increases linearly throughout the the duration of the outage.
Prikaži ovu nit -
47. You're bound by the CAP theorem much more often than you may think. Halting Problem's a bitch, too. 48. Eventual consistency doesn't help when the system you're debugging hasn't converged yet.
Prikaži ovu nit -
49. The source you're looking at is not the code running in production. 50. strace(1)/ktrace(1) doesn't lie. 51. Unless somebody's been playing LD_PRELOAD games.
Prikaži ovu nit -
52. Schrödinger's Backup -- "The condition of any backup is unknown until a restore is attempted." -- is overly optimistic. 53. There's an xkcd for the precise situation you find yourself in. (There's also one for at least half of these.)
Prikaži ovu nit -
54. At some point in your career you will implement half of kerberos. Poorly.
Prikaži ovu nit -
55. Any sufficiently successful product launch is indistinguishable from a DDoS; any sufficiently advanced user indistinguishable from an attacker. 56. Debugging any sufficiently complex open source product is indistinguishable from reverse engineering a black box.
Prikaži ovu nit - Još 20 drugih odgovora
Novi razgovor -
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.
: