I'm excited to share the first public notes about the project that and I have been working on for the past 6 months!
Building apps is too hard. We think that the solution involves *databases*.
... what?
Conversation
Very cool that you tested out DuckDB! You mentioned that the serialization / deserialization to and from the worker was a bottleneck for you. Did you happen to try using Apache Arrow alongside DuckDB? That should remove that serdes!
I love the idea of using a DB to track state!
1
1
I was really impressed with DuckDB when we tried it! I think we did experiment with Apache Arrow, but then gave up on it when we moved back to SQLite. It definitely feels like the right solution.
1
2
When we tried it, DuckDB was missing a couple of query optimizations that were really key for us, even though ti made almost all our queries faster (... the story of all query optimizer projects, I think).
Looking forward to giving it another spin some time!
1
1
I’d have to dig through some old notes to get all the details, but I think the big one was pushing a sort requirement into an index scan, particularly for retrieving a (small) range of data in the middle of the overall range, rather than doing a full sort or a very large top-N.