Transparent Encryption/Decryption Layer between LevelDB and Filesystem "Block Manager"

[CMCDragonkai]

Specification

Our current encryption/decryption layer sits on top of LevelDB. This causes problems for indexing #1 because when you want ot index something you'll need to expose keys, and keys have to be un-encrypted atm.

It may also increase performance of DB if encryption/decryption were operating at a block level rather at individual key-value level. It's the equivalent of using full-disk encryption and using leveldb on top.

We can't rely on OS provided full-disk encryption. So something that is in-between the current key-value DB like leveldb and the actual filesystem that is executed in JS or C++ would be needed.

There is a level-js which is a abstract-leveldown compliant store that can be wrapped in levelup. It is leveldb implemented in pure-JS which relies on IndexedDB. Currently IndexedDB doesn't exist natively on Node.js, but there are some implementations of it. This seems to give an opportunity to add a transparent encryption/decryption layer in between leveldb and IndexedDB.

Additional context

• Integrate Automatic Indexing into DB #1 (comment) - RocksDB at the C++ level has provision for incorporating encryption at a low level (not sure if this still key-value level or block level of abstraction)
• WIP: Integrating Automatic Indexing into DB #2 (comment) - discussion about level architecture in Node.js
• level is a library that bundles leveldown, level-js, levelup and encoding-down to create a single batteries-included package
  • the difference between leveldown and level-js is that leveldown uses the C++ leveldb library which only works in Node.js while level-js works on IndexedDB which exists in browsers
  • if IndexedDB were to exist in Node.js, one could use level-js as an isomorphic library that works on browsers and Node.js, not sure about NativeScript though
    • using IndexedDB could allow us to put a transparent encryption/decryption layer in between IndexedDB and level-js, thus enabling us flexible indexing, and probably better security
    • there are only 2 libraries providing IndexedDB in Node.js:
      • https://github.com/bigeasy/indexeddb
      • https://github.com/dumbmatter/fakeIndexedDB and FIDB persistence dumbmatter/fakeIndexedDB#12 (comment)
        • if fakeIndexedDB becomes realIndexedDB via leveldb, then this should be possible in NS as well, but you are doing something a bit funny: levelup API -> level-js -> transparent encryption/decryption -> IndexedDB -> leveldb or whatever, but this is sort of what happens in Chrome which implements IndexedDB using leveldb
        • see https://news.ycombinator.com/item?id=10188476 for discussion about IndexedDB in node.js
• Implement True Snapshot Isolation for LevelDB #4 - this may be impacted if IndexedDB is used
• https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20171220_encryption_at_rest.md - CockroachDB has done this already we can compare this

Tasks

1. - Investigate how level-js uses IndexedDB
2. - Attempt to implement or find a persistent IndexedDB, perhaps by being implemented by leveldb or sqlite, it seems like any performant implementation would have to use C++ at some point, also there are bunch of wrapper libraries, but not sure which ones actually perform real persistence
3. - Integrate this into PK

[CMCDragonkai]

Alternative is to instead work on leveldb directly and manipulate the C++ to allow one to plug in encryption/decryption.

[CMCDragonkai]

Along with IndexedDB, RocksDB, another option is lmdb based on the discussion here: 6107d9c#commitcomment-73269633.

The lmdb-js project already supports native encryption at the block level thus ensuring keys and values are encrypted.

[CMCDragonkai]

Since we are working at the C++ layer, this should mean we can finally attempt block level encryption. I wonder if we can just bind to node's openssl. https://github.com/nodejs/node-gyp/blob/master/docs/Linking-to-OpenSSL.md since it's already there.

Node's crypto and webcrypto API is likely built on top of the statically linked openssl. If we do the same, we would maintain parity with the crypto implementation. And it avoids bringing our own crypto library. Finally if we have to do, we can then do so with a native library rather needing it to be implemented in raw JS or web assembly.

[CMCDragonkai]

Relevant PRs:

• Provide a default block cipher implementation using AES. facebook/rocksdb#8943
• Encryption with Change Management facebook/rocksdb#7020
• Add support for compressor plugins facebook/rocksdb#6717