Today at a @qconnewyork AMA, @lauralifts asked a really good question "What do you think are the biggest opportunities in observability" and I've been thinking about it since. Curious what people think. I have mine ...
-
-
Temperature doesn't give you a macro signal; deltas and derivatives of temperature do. A hot day in SoCal is just normal, until it isn't, as is June gloom in June but not in August. I would visualize the system and then colorize it (separately) by delta-norm and rate of change.
-
One example from NOAA using GIF animation loop to simulate rate of change. https://www.cpc.ncep.noaa.gov/products/analysis_monitoring/enso_update/gsstanim.shtml …pic.twitter.com/MclAuWSO83
End of conversation
New conversation -
-
-
You are engineering at a completely different scale there. I think this is already happening though. Running a service fleet accepting a billion+ calls a day over 250 boxes we debug patterns that are not visible at the process level. This is akin to observing temp.
-
I detected the 2011 Japanese earthquake immediately long before any news. Not from any system failure but because online behavior patterns changed instantly at first shake and sent alarms screaming through macro observation. A whole country ceased clicking their mice at once.
- 2 more replies
New conversation -
-
-
And we could call it a “mood” and ask the system if its happy or sad. I wrote something on this that I need to post.... Our own human observability system is our emotions and our query and alerting system is called consciousness.
-
Reminds me of what Netflix reported in the context of Flux, the 'pain suit', letting you feel your infra. Time for some AR app, maybe?
End of conversation
New conversation -
-
-
A single scalar measurement is beyond my ken, but I could imagine something that was the max of several essential "temps" (latency, 5xx, network b/w, ...), and then it would be dreamy if I could click into that to see a system diagram showing me what edges were hot in what ways.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Is a combination of saturation and errors the answer here, perhaps?
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
A thought for you. I'm not that kind of mathematician, but apparently temperature is not mathematically well defined, so don't obsess too much about correctness.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
have you thought much about how to define the states for the components of your system?
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.