I wonder when this "databases are nearly always bottlenecked on IO" perception finally is going to die. I think it's been false for > 50% of instances for at least 15 years. And it's just plainly wrong when we can have small-ish servers with >16 internal NVMe drives.
-
Show this thread
-
Replying to @AndresFreundTec
Even more CPU used if DBMS does checksum and/or decompress after each storage read
1 reply 0 retweets 2 likes -
Replying to @MarkCallaghanDB
True. It's impressive though what kind of decompression bandwidth e.g. lz4 can have. It's feasible to reach >2GB/s even for some database uses (where typically the number of blocks decompressed at once is small-ish).
2 replies 0 retweets 1 like -
Replying to @AndresFreundTec @MarkCallaghanDB
Similar for checksums. Decent non-crypto hashes can reach > 4GB/s. And with hardware assist more is possible. The difficulty really is to have large enough runs to be checksummed/decompressed at once.
1 reply 0 retweets 1 like -
Replying to @AndresFreundTec @MarkCallaghanDB
Kinda curious now whether the crc32c instruction can benefit from pipelining to make it worthwhile to compute checksums for several blocks at once.
2 replies 0 retweets 0 likes -
Replying to @AndresFreundTec @MarkCallaghanDB
Agner lists a latency of three for r/r on skylake, and a reciprocal throughput of 1. Most database are probably using r/m, and there's multiple ports for loads. So I assume it'd help. PG only uses crc for the WAL however, and FNV1a for data. Data already uses SIMD for speed.
1 reply 0 retweets 1 like
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.