How to lose data: 1. A problem process eats disk space 2. Your email alert threshold is at 10% free 3. Your paging (wake me up) threshold is at 5% free 4. The ext4 reserved blocks are the default 5%. Woke up to FS at 5.01% free, a pile of <10% alert emails, and lost data.
-
Silent cronjob failures have broken my stuff too many times. I built an HTTP service which updates a key in redis with a timestamp, then use Prometheus to scrape matching keys, and alert if `now() - timestamp > threshold`. Run a curl in cron scripts on success, problem solved!
There are much easier ways of doing that. Either use the prometheus pushgateway, or use the textfile collector in node_exporter and just dump timestamps to files in your cronjobs. Been there done that :)
-
-
True! My biggest annoyance is the lack of real authentication in Pushgateway. If I have to run an authenticating reverse proxy, I may as well just write my own service and have client TLS or token based auth. node_exporter is great, but less feasible for Kubernetes CronJobs. :)
-
I already run nginx in front of so many things, it's almost always easier for me to do that. Super setut-dependant, I guess.
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.