Programmers / software engineers -- does this exist: I'm looking for a very efficient high-speed ID lookup system where I pass an integer to it (8 bytes, but configurable would be awesome) and it will index that id into file segments (configurable size) and periodically sort ...
Conversation
Replying to
... the file segments so that ID lookup is fast. In front of this might be a bloom filter to speed it up even more so that it doesn't always have to hit the disks to check if the ID exists.
I would need something that scales up to ~10 trillion ids.
1
2
Essentially, this would be a very low-level piece of software with an API that lets you insert an ID and/or check if an ID already exists. The files to hold the existing ids shouldn't really have much of an overhead if any.
2
2
Replying to
Maybe upscaledb is something you can use that may fit? upscaledb.com/index.html otherwise highly recommend reading blog posts / papers for more stuff like that lemire.me/blog/2012/09/1
Replying to
Take a look at Aerospike DB. Not sure if it fits your exact use case, but iirc, it was the fastest when it comes to this a year or two ago.
aerospike.com/benchmark/benc
Replying to
supports bloom filters out the box and is blazing fast. Might be worth a look.
1
2
Replying to
For testing membership, a 64-bit Judy1 array would also work.
1
2
Show replies