For example, Firefox has historically overused tiny SQLite transactions by committing every single change immediately, including doing commits with every letter that's typed in the address bar. They liked to blame SQLite when SQLite was doing exactly what they asked correctly.
Conversation
twitter.com/DanielMicay/st
Also, to clarify, you can safely leave out the final directory fsync as long as it's okay for either the previous version or the new version to be present after a crash / power loss. The fsync (or fdatasync) or the file data is what's needed for that.
Quote Tweet
Replying to @mik235 and @RichFelker
No, to do a transaction safely, you need to fsync the file to commit the data to storage, rename it and then fsync the containing directory to commit the transaction. By covering up the problem, ext4 is hurting performance while the bugs in software are still present and serious.
1
1
My understanding is that with data=ordered or at least with data=journal, just journaling should get you this either-or property with no fsync or fdatasync whatsoever. Is that wrong?
1
You definitely do need fdatasync, but fdatasync is traditionally a very fast operation compared to fsync. It's true that on ext3, data=ordered worked as you described, but there's no guarantee of that kind of coarse journal and ext4 changed this with delayed allocation of blocks.
1
Due to ext4 adding the auto_da_alloc hack, it will work without fdatasync in the default configuration since it forces the fsync itself, but it's not guaranteed to be configured that way and the hack doesn't exist on other modern file systems with delayed allocation like XFS.
1
I think it's terrible that ext4 added this hack because it encourages developers not to fix their code to use actual transactions. There's a pervasive mistake of using fsync on the file after the rename instead of before the rename which is broken and can leave inconsistent data.
1
IIRC the problem was that people were doing fsync/rename backwards and not fixing their code, even after wide-spread reports of data loss.
Presumably because previous filesystems happened to get away with it?
I guess slow is better than lost data?
1
It's not slow to do a properly scoped fdatasync before rename. It's faster than doing a full fsync on the file after rename, which doesn't even guarantee that the rename was synced because doing that requires an fsync on the directory rather than the file anyway.
2
By doing rename before sync, it's possible the rename is committed to storage before data. On ext3, it wasn't possible with data=ordered because it didn't have delayed allocation of blocks, but that was never guaranteed or portable. Doubt developers knew about that guarantee.
2
The issue is really that developers are not actually thinking about doing a transaction. It's very straightforward to do it correctly as long as you realize file names are part of directory metadata and use fsync / fdatasync based on their semantics not implementations like ext3.
1
I think the main issue is that developers add fsync as an afterthought rather than trying to provide a real transaction with guaranteed consistency. I'm not really convinced developers thought running fsync after rename guaranteed consistency but rather they just sprinkled it on.