CRDTs are a useful piece of the puzzle, but they don’t solve the problem as fully as people often imagine. The I&S folks are working diligently on many of these. But one problem I don’t know how to solve is the complexity of the resulting opaque file formats.
Conversation
In an ideal world, everything’s just plaintext, so you can modify it with whatever tool you like. No libraries needed. Problems with this include semantic sync, indexing, structured data, etc. SQLite is almost as good, if the schema is well-defined: sqlite.org/appfileformat.
1
30
Maybe someday automerge’s serialization format will be as durable and universal as SQLite’s, but in the meantime, it’s certainly not amenable to casual concurrent local access in some user script.
1
1
18
Of course, SQLite leaves you to solve syncing. The typical approach is a write-only log of events (CRDT mutations or otherwise) which can be replayed across replicas. Snapshots of entity states are computed from queries across these events.
1
17
That’s the approach Orbit takes: simple event structures (simpler than CRDTs, sacrificing some of their key guarantees to reduce complexity), with well-defined merge operations, in a SQLite db. Users can write scripts to insert their own events if they like.
2
1
24
I’m still not satisfied with this. Reading/writing data yourself requires using Orbit’s libraries or a SQLite library and an understanding of the schema. I’d like to just expose the data as plaintext, but I’m not yet sure how to achieve that practically.
3
21
One approach is to define Git-style “plumbing” CLI primitives which offer plaintext-like I/O to your data structures. Another is to define a FuseFS layer, style.
1
13
Switching gears: another hard thing about implementing this type of data format is that web browsers demand their own implementation. kindly implemented an IDB-based Orbit backend. Excited about absurd-sql here:
1
19
Orbit’s cloud server has its own (third, ugh) backend implementation, which it uses to offer APIs, aggregate analytics (for my research), and services like study reminder notifications.
1
12
I see now why people don’t generally do this. It was a huge amount of work. I’m honestly pretty frustrated with myself for shaving this yak: too much time polishing my telescope, too little time looking at stars.
1
3
51
As it happens, Orbit’s implementations are quite general (i.e. they have almost no specific knowledge of Orbit’s structures), so perhaps they’ll be of use to others. See store-*, sync, and backend packages here:
An in between solution is to keep source of truth on the server but a) guarantee all data can import/export to a simple format like csv, and b) periodically export it to user owned place (eg s3, gdrive) so the user doesnt worry that “a” is a fake claim.
Great report from the front lines—thanks for sharing!
One observation that makes is that web browsers aren't a good fit for local-first. They're fundamentally thin clients with throwaway local caches. Maybe that's part of the mismatch for Orbit?
3
11
Thick clients with a sync-style model like Dropbox, Git, Craft, and Obsidian are easier to fit with this, although often still don't have a user-facing flat file format outside of explicit export.
2
3
Show replies
Replying to
For iOS devs, proper iCloud Documents support means merging concurrent edits. I feel like I walked a similar, simpler road to what you describe with Orbit and landed in an analogous place for a different software ecosystem. Just wrote my thoughts here: bdewey.com/til/2021/08/21/
1
1
Currently I’m staying far away from automatic merging of text edits precisely because of the “opaque file format” problem. But automatic merging of edits made to “conceptually different text chunks” is straightforward in a replicated key/value store & the format is still simple.
1