The troubles are well articulated here: inkandswitch.com/local-first.ht
I recently finished re-architecting to expose an on-disk format, thought I’d write a bit about what I learned.
Conversation
It’s hard to pull this off, as I&S describe. You need to design a file format which syncs efficiently and reliably, which is already tough. Gets tougher when your SaaS is an active replica with its own public API, and when you layer on indexing/querying, migration, sparsity.
1
29
There are enterprise solutions which solve these problems (Realm, CouchDB), but one consistent limitation is that they “own” the file format, and often the server component as well. I don’t feel good exposing my on-disk format to local clients if it’s tied to a specific backend.
1
17
CRDTs are a useful piece of the puzzle, but they don’t solve the problem as fully as people often imagine. The I&S folks are working diligently on many of these. But one problem I don’t know how to solve is the complexity of the resulting opaque file formats.
1
30
In an ideal world, everything’s just plaintext, so you can modify it with whatever tool you like. No libraries needed. Problems with this include semantic sync, indexing, structured data, etc. SQLite is almost as good, if the schema is well-defined: sqlite.org/appfileformat.
1
30
Maybe someday automerge’s serialization format will be as durable and universal as SQLite’s, but in the meantime, it’s certainly not amenable to casual concurrent local access in some user script.
1
1
18
Of course, SQLite leaves you to solve syncing. The typical approach is a write-only log of events (CRDT mutations or otherwise) which can be replayed across replicas. Snapshots of entity states are computed from queries across these events.
1
17
That’s the approach Orbit takes: simple event structures (simpler than CRDTs, sacrificing some of their key guarantees to reduce complexity), with well-defined merge operations, in a SQLite db. Users can write scripts to insert their own events if they like.
2
1
24
I’m still not satisfied with this. Reading/writing data yourself requires using Orbit’s libraries or a SQLite library and an understanding of the schema. I’d like to just expose the data as plaintext, but I’m not yet sure how to achieve that practically.
3
21
(part of me thinks that maybe we /should/ all just get really fluent with SQLite & set a cultural expectation that everyone is as comfortable with that as they are with plaintext today)
2
1
8
I think that a good part of the cultural obsession with plaintext comes from decades of bad SQL tools. A SQLite database and an embedded schema *should* be better than a blob of JSON, but somehow it often isn't.
Right. There still isn't a SQL editor for Mac that I'd actually enjoy using for some extended creative work. Compare that to, say, text editors, which get lavishly appointed…
3
2
(But, like, to be clear: I wouldn't enjoy editing JSON as part of extended creative work either! It's really just true plaintext and things very close to it that get away with this…)
1
1
Show replies