Our ZFS space-eating problem turned out to be references to now-deleted files held by the (kernel) nlockmgr. Restarting it cured things.
In today's fun #ZFS experience, we appear to have a filesystem on which deleting things doesn't free up space. No snapshots/etc exist.
-
-
- View other replies
-
As the file references were held in some way inside the OmniOS kernel, fuser didn't show them. Real total observability is important.
-
@thatcks How did you spot it? -
@oclsc I asked the Illumos ZFS developers mailing list & the people at Delphix who'd hit this told me what magic zdb things would be signs. -
@thatcks I keep misreading that as `illustrious ZFS developers.'
-
-
-
@thatcks something holding open the file handle? -
@gpapilion No filehandles, no sign of anything. The FS turns over about 5 GB a week (new files added, old deleted) & has been losing that.
-
-
-
-
@thatcks i see though it was an open filehandle. my bad
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.
Chris Siebenmann
(((Norman Wilson)))
Geoffrey Papilion
smerz