Someone added a few thousand entries to a list that lets anyone append to it.
GnuPG, software supposed to defeat state actors, suddenly takes minutes to process entries.
How big is that list you ask? 17 MiB. Not GiB, 17 MiB. Like a large picture.
dev.gnupg.org/T4592
Conversation
I made multi-process transactions on SQLite databases of 600GiB, so I have no idea what they are talking about, but let's say I would trust SQLite more than, uh, whatever is currently failing at 17MiB.
4
8
77
Replying to
SQLite has supported the WAL journal mode for 9 years which is generally a better choice for any internal usage rather than usage as a file format.
sqlite.org/wal.html
Not sure why they're claiming that it requires making a copy of the database for transactions. It's 2019.
1
6
WAL journal mode also supports any number of concurrent readers with 1 concurrent writer. WAL makes it a great fit in many cases where you previously would have needed a database server. SQLite scales very well to fairly large amounts of data and just can't do concurrent writers.
1
1
Replying to
But even a rollback journal is not a copy of the whole database! WAL has better locking semantics, but even plain DELETE mode would work here.
Also, it does serialization of concurrent writes, so unless you have a lot of parallelizable writes (not the case), it's not a problem.
1
Replying to
Yeah, it's only a copy of the pages that are being changed. It's sized based on the size of the database for efficiency, but it doesn't need to copy everything for each transaction. The traditional locking can definitely be an issue but WAL mode makes it viable in far more cases.
1
1
Readers and writers blocking each other is far more of an issue than writers blocking each other, since you can end up in situation where writes are never happening at all, and readers also tend to be far more numerous. Probably not a good niche for SQLite in 2009, but it's 2019.
1
Replying to
Agreed on WAL being the better option most of the time, but does TRUNCATE mode really make the journal as big as the db every time? It seems like a weird choice as no client ordinarily reads the journal, and I can't find that documented anywhere.
1
Replying to
It contains the previous copies of the pages modified by the transaction, so for the DELETE and TRUNCATE cases, it wouldn't need to end up larger than the transaction.
sqlite.org/fileformat.htm
I was thinking about PERSIST mode, which generally ends up as large as the database.
2
Replying to
PERSIST ends up as large as the largest transaction ever, or journal_size_limit, whichever smaller.
Pretty sure SQLite would be strictly better in basically any mode than anything ad-hoc.
Sigh.
1
Replying to
Yeah, no doubt about that. It's definitely wrong to say that the traditional journal mode copies the entire database for every transaction. I can see why people would have that misunderstanding though because it's true you can need up to that much space for the rollback journal.