Programmers / software engineers -- does this exist: I'm looking for a very efficient high-speed ID lookup system where I pass an integer to it (8 bytes, but configurable would be awesome) and it will index that id into file segments (configurable size) and periodically sort ...
Conversation
... the file segments so that ID lookup is fast. In front of this might be a bloom filter to speed it up even more so that it doesn't always have to hit the disks to check if the ID exists.
I would need something that scales up to ~10 trillion ids.
Replying to
Essentially, this would be a very low-level piece of software with an API that lets you insert an ID and/or check if an ID already exists. The files to hold the existing ids shouldn't really have much of an overhead if any.
2
2