Is there an "ohshitohfuck" filesystem, that starts moving the least-surprising data over to another drive, or compressing data when that happens? Ideally, moving it all back later when space frees.
How to lose data: 1. A problem process eats disk space 2. Your email alert threshold is at 10% free 3. Your paging (wake me up) threshold is at 5% free 4. The ext4 reserved blocks are the default 5%. Woke up to FS at 5.01% free, a pile of <10% alert emails, and lost data.
-
-
-
So what caused this problem was actually the cronjob in charge of copying data to another filesystem and deleting old data not working...
- Show replies
New conversation -
-
-
Oof :/
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
What do you use for paging? Not sure I want more than alertmanager emails, but I'm curious what's being used nowadays.
-
I use Pushover, which is nice and simple, one time app purchase fee, integrated with Alertmanager. Recommended for personal use.
End of conversation
New conversation -
-
-
The best part is that if you accidentally fill up the root file system, random tools and things like path completion just stop working, so recovering it sucks.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.