Did you know ... Search Documentation:
Packs (add-ons) for SWI-Prolog

Package "rocksdb"

Title:SWI-Prolog interface to RocksDB
Rating:Not rated. Create the first rating!
Latest version:0.14.3
SHA1 sum:c59049014cf7ffdb57b4b9cd87c8bdb7273eb2b3
Author:Jan Wielemaker <jan@swi-prolog.org>
Maintainer:Jan Wielemaker <jan@swi-prolog.org>
Packager:Jan Wielemaker <jan@swi-prolog.org>
Home page:https://github.com/JanWielemaker/rocksdb
Download URL:https://github.com/JanWielemaker/rocksdb.git

Reviews

No reviews. Create the first review!.

Details by download location

VersionSHA1#DownloadsURL
0.5.0e92e951173fbafa70cb1225941e672ddc98560dc3https://github.com/JanWielemaker/rocksdb/archive/V0.5.0.zip
0.6.0687ab5156fb4e05b7eff71bfc6df602c343c5fb110https://github.com/JanWielemaker/rocksdb/archive/V0.6.0.zip
0.7.01875de6403e2606a9040a591bfffefe9d34dfe5022https://api.github.com/repos/JanWielemaker/rocksdb/zipball/V0.7.0
c71a05346e46054591006bd61b6a0201feda91911https://github.com/JanWielemaker/rocksdb.git
0.8.098ec2151732a03ca666ce1c0196b850b370138b82https://github.com/JanWielemaker/rocksdb.git
9a88bb9ab4c657d305b5cb00bffb708304e866d013https://github.com/JanWielemaker/rocksdb.git
f110766ee97cfbc6fddd4c33b7238f00e76ecc1826https://github.com/JanWielemaker/rocksdb.git
ff3ddc768e84d51aca375bfabe02234d0596485e4https://github.com/JanWielemaker/rocksdb.git
0.9.05034a96229e2e7d366718ccb200537e6ab6723a810https://github.com/JanWielemaker/rocksdb.git
0.10.0e253458069c7608e8d10a416c79c7e5ed5fac1d915https://github.com/JanWielemaker/rocksdb.git
0.12.092ea37233ff6487372d902e75174c86f953cea7312https://github.com/JanWielemaker/rocksdb.git
e6caf9851904b95bd56f900bd63b42f31f746e641https://github.com/JanWielemaker/rocksdb.git
0.14.3c59049014cf7ffdb57b4b9cd87c8bdb7273eb2b31https://github.com/JanWielemaker/rocksdb.git

SWI-Prolog interface for RocksDB

This is a SWI-Prolog pack that provides library(rocksdb), a binding to RocksDB.

Installation

The installation requires a recent C++ compiler. RocksDB can work with several compression libraries. Most systems have zlib installed, but others may provide better performance or less resource usage. See INSTALL.md. On Ubuntu 22.04 the following packages were installed:

sudo apt install libsnappy-dev liblz4-dev libzstd-dev libgflags-dev

Once these are in place, a simple

?- pack_install(rocksdb).

Should do the trick. Note that this clones RocksDB and builds it the way we need the library. This requires significant disk space (1.4Gb) and takes long (several minutes on a modern machine).

This code depends on version 2 of the SWI-cpp2.h file that is part of SWI-Prolog version 9.1.0 and later.

Why are you not using the pre-built librocksdb?

There are a number of issues with several pre-built versions of librocksdb:

  • Shared objects are often linked to jemalloc or tcmalloc. This prevents lazy loading of the library, causing either problems loading or running the embedded rocksdb.
  • Various libraries are compiled without RTTI (RunTime Type Info), which breaks subclassing RocksDB classes.
  • Static library is by default compiled without -fPIC and thus not usable.

As is, the most reliable way around is to include RocksDB, so we can control the version and build it the way that best fits our needs: as a static library with RTTI and -fPIC.

Manual installation

If the above fails

  • Clone this prepository in the pack directory of your installation or clone it elsewhere and link it.
  • Run ?- pack_rebuild(rocksdb). to rebuild it. On failure, adjust Makefile to suit your installation and re-run the pack_rebuild/1 command.

Status

The wrapper provides most of functionality of RocksDB. Column families are not supported, nor are the following features (this is not an exhaustive list):

  • user-defined timestamp
  • snapshots
  • slices
  • table filter
  • rate limiter
  • DB paths
  • event listeners
  • user-defined cache
  • user-defined checksums
  • user-defined statistics handler
  • user-defined logger
  • loading options from a file

Options, logging and statistics

The options are defined and documented in file rocksdb/include/rocksdb/options.h.

RocksDB persists its options to an OPTIONS file but there currently is no Prolog support loading this, as in rocksdb/examples/options_file_example.cc.

RocksDB provides extensive logging and statistics for adjusting the options. These are normally written to the LOG file(s) in the database directory, but can be written elsewhere by setting the db_log_dir option.

Statistics can be turned on by the statistics(true) option. Additional statistics-related options are:

  • stats_dump_period_sec
  • stats_persist_period_sec
  • persist_stats_to_disk
  • stats_history_buffer_size

See also https://github.com/facebook/rocksdb/wiki/Statistics http://rocksdb.org/blog/2018/08/01/rocksdb-tuning-advisor.html https://github.com/EighteenZi/rocksdb_wiki/blob/master/RocksDB-Tuning-Guide.md

Implementation notes

If you want to support user-defined statsitics callbacks (DBOptions.info_log), it appears that there are multi-threaded issues -- a simple implementation that called `Sfprintf(Suser_error, ...)` git a SEGV error when `Flush()` was implemented using `Sflush(Suser_error)`. For reason reason, we don't support `info_log(:Goal) or info_log_flush(:Goal)' in the optins. Setting `info_log_level results in LOG files in the RocksDB directory, which is probably good enough.

Other callback implementations (such as rate_limiter, statistics, listeners) could have similar problems.

TODO

These seem to be the most useful missing features that don't require a lot of work:

Contents of pack "rocksdb"

Pack contains 10 files holding a total of 92.9K bytes.