How to lose data: 1. A problem process eats disk space 2. Your email alert threshold is at 10% free 3. Your paging (wake me up) threshold is at 5% free 4. The ext4 reserved blocks are the default 5%. Woke up to FS at 5.01% free, a pile of <10% alert emails, and lost data.
-
Is there an "ohshitohfuck" filesystem, that starts moving the least-surprising data over to another drive, or compressing data when that happens? Ideally, moving it all back later when space frees.
So what caused this problem was actually the cronjob in charge of copying data to another filesystem and deleting old data not working...
-
-
oof! Only suggestion I've gotten about this on linux machines is to use separate mounts for each application, so at least it can't kill the box. :/ Failures from that kind of cronjob should probably be higher priority. Maybe after a few in a row. Like server 911, or something.
-
This *was* a separate mount, so it indeed only killed the app. But the app was storing data that only comes in once, so several hours of data are now lost.
- Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.