CRDTs are a useful piece of the puzzle, but they don’t solve the problem as fully as people often imagine. The I&S folks are working diligently on many of these. But one problem I don’t know how to solve is the complexity of the resulting opaque file formats.
Conversation
In an ideal world, everything’s just plaintext, so you can modify it with whatever tool you like. No libraries needed. Problems with this include semantic sync, indexing, structured data, etc. SQLite is almost as good, if the schema is well-defined: sqlite.org/appfileformat.
1
30
Maybe someday automerge’s serialization format will be as durable and universal as SQLite’s, but in the meantime, it’s certainly not amenable to casual concurrent local access in some user script.
1
1
18
Of course, SQLite leaves you to solve syncing. The typical approach is a write-only log of events (CRDT mutations or otherwise) which can be replayed across replicas. Snapshots of entity states are computed from queries across these events.
1
17
That’s the approach Orbit takes: simple event structures (simpler than CRDTs, sacrificing some of their key guarantees to reduce complexity), with well-defined merge operations, in a SQLite db. Users can write scripts to insert their own events if they like.
2
1
24
I’m still not satisfied with this. Reading/writing data yourself requires using Orbit’s libraries or a SQLite library and an understanding of the schema. I’d like to just expose the data as plaintext, but I’m not yet sure how to achieve that practically.
3
21
Did you consider an approach similar to iTunes?
- client-side text file that users can modify
- client-side opaque binary format that’s derived from the text file at startup
- server-side format that’s derived from events that change the client-side opaque binary format
1
Yes! I think this is a pretty compelling approach, and I may build a text-based frontend to the existing SQLite-based format like this. I realized that I'd need the latter first no matter wehat, so I started there.
1
I think has backed into something like this with his suite of tools that convert to and from sqlite: github.com/simonw/sqlite- as well as related formats.
1
1
I did an experiment yesterday that you might find interesting: using SQLite triggers to maintain a table tracking when each row in a table was most recently modified or deleted
2
5
Nice! This is a nice general solution for systems which can tolerate something like a "last-writer-wins" policy for conflict resolution.