One good side-effect from the PostgreSQL trademark issue, I learned how to (and did) donate to PostgreSQL.
postgresql.org/about/donate_p
Conversation
Alas, there wasn't an option to target that to the development of a write-optimized storage engine for PG.
3
4
Replying to
FWIW the startup that I just joined (Zenith) is building an open source Postgres extension that uses log-structured storage. The design is much closer to Aurora than MyRocks, though - efficiency and write amp are not the main goal. Changes very little within the standard AMs.
2
1
7
Replying to
Agreed. But the model is also theoretically appealing. The Fineline paper author sometimes illustrates the problem with checkpoints using "the faucet analogy" -- he paints a pictures that's all too familiar to me. See "Fig. 5: Dirty page backlog" from: dl.gi.de/bitstream/hand
1
3
Replying to
Nice paper, have been there too, now you trade one hard problem (managing writeback rate) for another. With a b-tree there is natural feedback when buffer pool is too dirty. With log-structured you usually lose that natural feedback, need new ways to find balance.
1
Replying to
Natural feedback is essential, but I suspect that it's not as effective as one would hope in Postgres/InnoDB. With Aurora you can have many WAL records that you may need to replay, without it necessarily mattering at all. Checkpoints create *systemic* risk, forcing us to go slow.
2
Replying to
The question I have is given a similar amount of HW:
1) What are read, space & write-amp for b-tree vs this approach?
2) What are other side-effects. b-trees suffer from stalls if checkpoint can't keep up, does this other approach have problems?
I don't expect an answer today.
2
Replying to
Unsure, because it both helps and hurts WA. I'll just say this: the design exploits natural feedback in a way that seems old and new. On-demand, page-at-a-time model figures out which pages to materialize based on practical ideas about debt. Enormous page-level skew is common.