Rocksdb log file disk consumption issue

[vishal-groww]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Expected behavior

.log files are deleted once they are processed

Actual behavior

.log files are accumulating in the rocksdb data directory

Steps to reproduce the behavior

We are using rocksdbjni version 9.0.0
We are mostly using default settings of RocksDB. Below is the piece of code that creates transaction db. Can you please let me know how to configure the DBOptions so that log files are flushed once their processing is done?

TransactionDB db;
try (DBOptions dbOptions = new DBOptions().setCreateIfMissing(true)
.setCreateMissingColumnFamilies(true);
TransactionDBOptions transactionDBOptions = new TransactionDBOptions()) {
db = TransactionDB.open(
dbOptions,
transactionDBOptions,
pathForRocksDb,
columnFamilyDescriptors,
columnFamilyHandles);
String columnFamilyNameString = String.join(",", getColumnFamilyList());
db.put("COLUMN_FAMILY_NAMES_KEY".getBytes(), columnFamilyNameString.getBytes());
} catch (Exception e) {
log.error("Exception while opening transaction db ", e);

I tried overriding the compaction setting, but that did not work

try (final ColumnFamilyOptions cfo = new ColumnFamilyOptions()) {
org.rocksdb.SkipListMemTableConfig config = new org.rocksdb.SkipListMemTableConfig();
// cfo.memTableFactoryName()
config.setLookahead(1);
cfo.setCompactionStyle(UNIVERSAL);
cfo.setMemTableConfig(config);
cfo.setMemtableMaxRangeDeletions(1);
columnFamilyDescriptors = columnFamiliesToOpen.stream()
.map(name -> new ColumnFamilyDescriptor(name.getBytes(), cfo))
.collect(Collectors.toList());
}

When I starting the java app, i am getting the exception "Message: Missing configurable object: memtable_factory" with the above change

[hx235]

.log are WAL files. I don't see why overriding compaction setting work.

Can you check your max_total_wal_size, WAL_ttl_seconds and WAL_size_limit_MB value in options.h

Do you by any chance use DisableFileDeletions()?