Thanks to nexern! He recommended both nanomsg and sophia and they are pure C projects that I feel right at home in.
I am especially liking Sophia
http://sphia.org/:
"Sophia is a modern embeddable key-value database.
It has unique architecture that was created as a result of research and reconsideration primary algorithmic constraints of Log-file based data structures, such as LSM-tree. (see architecture)
Sophia is designed for fast write (append-only) and read (range query-optimized, adaptive architecture) small to medium-sized key-values.
Sophia is feature-rich (see features).
BSD licensed and implemented as small C-written library with zero dependencies."
"Sophia database and its architecture was born as a result of research and reconsideration of primary alghorithmic constraints that relate to growing popular Log-file based data structures, such as LSM-tree, B-tree, etc.
Most Log-based databases tend to organize own file storage as a collection of sorted files which are periodically merged. Thus, without applying some key filtering scheme (like Bloom-filter) in order to find a single key, database has to traverse all files that can take up to O(files_count * log(file_key_count)) in the worst case, and it's getting even worse for range scans, because Bloom-filter is incapable to operate with key order.
Sophia was designed to improve this situation by providing faster read while still getting benefit from append-only design."
"Sophia defines a small set of basic operations which can be applied to any database object. Configuration, Control, Transactions and other objects are accessible using the same methods. Methods are called depending on used objects. Methods semantic may slightly change depending on used object."
The API to Sophia is very similar to a JSON API, so mapping it to JSON API only has to deal with the issue of pointers. For now I will limit it to built in plugin mode as then pointers can actually be used. To allow usage by other processes (and nodes) I will need to add a layer that maintains the various pointers and objects for a caller, but this is a tangent so no time for it now. maybe somebody else would want to do this?
"All methods are thread-safe and atomic."
That simple statement has a LOT of implications, all of them good.
"It is possible to use compression for a specified databases using db.database_name.compression.
Supported compression values: lz4, zstd, none (default)."
"Sophia supports single-statement and multi-statement transactions."
"There are no limit on a number of concurrent transactions. Any number of databases can be involved in a multi-statement transaction."
"Snapshots represent Point-in-Time read-only database view.
It is possible to do sp_get(3) or sp_cursor(3) on snapshot object. To create a snapshot, new snapshot name should be set to snapshot control namespace."
"Sophia supports asynchronous Hot/Online Backups.
Each backup iteration creates exact copy of environment, then assigns backup sequential number.
...
Procedure call is fast and does not block."
"It is possible to start incremental asynchronous checkpointing process, which will force branch creation and memory freeing for every node in-memory index. Once a memory index log is free, files also will be automatically garbage-collected."
"Database monitoring is possible by getting current dynamic statistics via sp_ctl(3) object.
Also it is possible to get current memory usage or trace every worker thread (scheduler namespace). Database indexes have total number of keys and transactional duplicates (MVCC). Total number of nodes is node_count. Branches distribution can be obtained via branch_count, branch_avg, branch_max, histogram_branch"
From what I can tell sophia has all the features of BDB, just without the bloat. maybe there are some advanced things in BDB like setting up as a synchronized cluster, but I couldnt figure out how to do that quickly. With sophia, it has compact API and best of all nice C code so I can always debug it if there are any bugs (I doubt there will be any meaningful bugs)
To build it you probably wont believe this, but it creates a single source file that includes all the other files! Where have I seen that before?
http://sphia.org/v12.html
I dont know about you, but I usually dont get enthusiastic over a key/value store system. With sophia, well, if it does what it says it does, it will be able to be used for pretty much all the DB needs I can foresee for the medium term
James