Conversation

Replying to

and

twitter.com/DanielMicay/st Also, to clarify, you can safely leave out the final directory fsync as long as it's okay for either the previous version or the new version to be present after a crash / power loss. The fsync (or fdatasync) or the file data is what's needed for that.

Quote Tweet

Daniel Micay

@DanielMicay

Jul 15, 2019

Replying to @mik235 and @RichFelker

No, to do a transaction safely, you need to fsync the file to commit the data to storage, rename it and then fsync the containing directory to commit the transaction. By covering up the problem, ext4 is hurting performance while the bugs in software are still present and serious.

Replying to

and

My understanding is that with data=ordered or at least with data=journal, just journaling should get you this either-or property with no fsync or fdatasync whatsoever. Is that wrong?

Replying to

and

You definitely do need fdatasync, but fdatasync is traditionally a very fast operation compared to fsync. It's true that on ext3, data=ordered worked as you described, but there's no guarantee of that kind of coarse journal and ext4 changed this with delayed allocation of blocks.

Replying to

and

Due to ext4 adding the auto_da_alloc hack, it will work without fdatasync in the default configuration since it forces the fsync itself, but it's not guaranteed to be configured that way and the hack doesn't exist on other modern file systems with delayed allocation like XFS.

Replying to

and

I think it's terrible that ext4 added this hack because it encourages developers not to fix their code to use actual transactions. There's a pervasive mistake of using fsync on the file after the rename instead of before the rename which is broken and can leave inconsistent data.

Replying to

and

IIRC the problem was that people were doing fsync/rename backwards and not fixing their code, even after wide-spread reports of data loss. Presumably because previous filesystems happened to get away with it? I guess slow is better than lost data?

Replying to

and

It's not slow to do a properly scoped fdatasync before rename. It's faster than doing a full fsync on the file after rename, which doesn't even guarantee that the rename was synced because doing that requires an fsync on the directory rather than the file anyway.

Replying to

and

By doing rename before sync, it's possible the rename is committed to storage before data. On ext3, it wasn't possible with data=ordered because it didn't have delayed allocation of blocks, but that was never guaranteed or portable. Doubt developers knew about that guarantee.

Replying to

and

I disagree with this assessment. It was guaranteed by what my, and I think many others', understanding of what data=ordered meant. Delayed allocation is a hack that licenses the fs driver to violate what data=ordered means.

Replying to

and

It's only relevant what data=ordered means if the code was specifically written for ext3 rather than being portable across filesystems. It shouldn't matter what data=ordered means for the vast majority of code not written to target a specific filesystem implementation.

Replying to

and

The parameter doesn't exist at all for other filesystems like xfs or other operating systems. Using fdata and fdatasync per the actual semantics is straightforward and correct code doesn't depend on the ext4 hacks to inject fsync based on non-portable, non-guaranteed heuristics.

6:57 AM · Jul 15, 2019Twitter Web Client