One good side-effect from the PostgreSQL trademark issue, I learned how to (and did) donate to PostgreSQL.
postgresql.org/about/donate_p
Conversation
Alas, there wasn't an option to target that to the development of a write-optimized storage engine for PG.
3
4
Replying to
FWIW the startup that I just joined (Zenith) is building an open source Postgres extension that uses log-structured storage. The design is much closer to Aurora than MyRocks, though - efficiency and write amp are not the main goal. Changes very little within the standard AMs.
2
1
7
Replying to
Agreed. But the model is also theoretically appealing. The Fineline paper author sometimes illustrates the problem with checkpoints using "the faucet analogy" -- he paints a pictures that's all too familiar to me. See "Fig. 5: Dirty page backlog" from: dl.gi.de/bitstream/hand
1
3
Replying to
Nice paper, have been there too, now you trade one hard problem (managing writeback rate) for another. With a b-tree there is natural feedback when buffer pool is too dirty. With log-structured you usually lose that natural feedback, need new ways to find balance.
1
Replying to
Natural feedback is essential, but I suspect that it's not as effective as one would hope in Postgres/InnoDB. With Aurora you can have many WAL records that you may need to replay, without it necessarily mattering at all. Checkpoints create *systemic* risk, forcing us to go slow.
My work on indexing built on the observation that even when the system seriously misbehaves, most individual pages look just fine, most of the time. So all we need to do is make them look fine *all* of the time (kinda). "The whole is not the sum of its parts" -- applies here too.
1
Replying to
The question I have is given a similar amount of HW:
1) What are read, space & write-amp for b-tree vs this approach?
2) What are other side-effects. b-trees suffer from stalls if checkpoint can't keep up, does this other approach have problems?
I don't expect an answer today.
2
I assume the approach you mention has many other benefits, include less chance of stalls from checkpoint. Even if it has worse efficiency, that might not matter to anyone.
1
Show replies