Use `bufferWrap` to support `ArrayBuffer`, `Uint8Array` and `Buffer`

[CMCDragonkai]

Specification

The ArrayBuffer type is more generalised in the JS ecosystem. The DB right now focuses on taking Node Buffers.

Node buffers can be converted to ArrayBuffer easily, and ArrayBuffer can be wrapped in Node buffers.

According to this comment: Level/level-js#34 (comment) it is possible to use ArrayBuffers directly in leveldb, it just not documented.

It says that the asBuffer has to be turned false.

I'm not sure how this would work with keyEncoding and valueEncoding that we have set to 'binary'.

Our raw is also specifying that we would get Buffer, would that mean we instead say that we return ArrayBuffer instead, and with keys it would be string | ArrayBuffer?

Furthermore this would impact js-encryptedfs, so it's worthwhile to explore the implications of this.

The primary reason to do this would be cross-platform compatibility for mobile platforms that may not have the Node buffer.

The alternative would be use https://github.com/feross/buffer everywhere as a dependency so that way everything just uses the buffer library. This will mean that any usage of import { Buffer } from 'buffer'; will be resolved by feross/buffer first though, so one should beware of that.

Note that all Node Buffer is Uint8Array which is ArrayBuffer.

If something supports ArrayBuffer, they would support Uint8Array and Buffer at the same time.

One benefit would be integration of js-id MatrixAI/js-id#1 can be simplified since Id as Uint8Array can be stored directly in the DB without first wrapping them as Buffer.

Additional context

• Array Buffers Level/level-js#34 (comment)

Tasks

1. - Investigate ArrayBuffer compatibility in leveldb
2. - Investigate to what extent can we enable compatibility with ArrayBuffer in DB without losing Buffer support
3. - Replace all uses of string | Buffer with string | ArrayBuffer for keys, but retain Buffer on returned output.
   • fix domainPath with Use bufferWrap to support ArrayBuffer, Uint8Array and Buffer #3 (comment)
   • this incorporates the Robustness principle: https://en.wikipedia.org/wiki/Robustness_principle
4. - Change this.crypto.key to be assumed to be ArrayBuffer
5. - Change toArrayBuffer to an explicit slice copy of all possible instances of ArrayBuffer type
6. - The fromArrayBuffer is still useful since we return Buffer.
7. - Change serialize and deserialize to use ArrayBuffer and TextEncoder and TextDecoder which can do the utf-8 encoding/decoding.
8. - Check if PK is using any of the utility functions here, some uses of deserialize and serialize might need to change.
9. - Ensure that most signatures are still compatible
10. - Update all tests to include Uint8Array and ArrayBuffer usage, and ensure that slice-copying is working.
11. - Check that there's no performance regression on the benchmarks

[CMCDragonkai]

Note that node module resolution would load locally first before loading from anywhere else.

I'm not sure how this applies to non-node runtimes like in NativeScript. Recommend not doing anything here until we have understood the platform properly.

The usage of ArrayBuffer would be a standardisation across other JS libraries and anything claiming to support ecmascript remember.

[CMCDragonkai]

Note that all Node Buffer is Uint8Array which is ArrayBuffer.

If something supports ArrayBuffer, they would support Uint8Array and Buffer at the same time.

We are also using alot of Uint8Array where Buffer was normally used. But ArrayBuffer is always useful from a "storage" POV. And returning Uint8Array means you always returning something that can be ArrayBuffer too either by using it directly or with bytes.buffer.

Therefore we can be satisfied with supporting Uint8Array as well. But maximum compatibility is going to be the ArrayBuffer.

[CMCDragonkai]

The type we use in js-db is LevelDB<string | Buffer, Buffer>. This means the values returned satisfy the interface of Buffer, Uint8Array and ArrayBuffer all together.

As for string | Buffer in src/DB.ts, if it works just by changing Buffer to ArrayBuffer, then this would solve this issue, and one could pass Buffer, ArrayBuffer or Uint8Array as the key.

[CMCDragonkai]

The only issue would be the domainPath that uses prefixer. Does that work with just ArrayBuffer, or would that need Uint8Array?

[CMCDragonkai]

No it doesn't because it uses Buffer.isBuffer to check. It throws this error Error: key should be a string or a buffer if it is not Buffer.

However one could wrap it in our domainPath function. We should be able to zero-copy it into a Node Buffer.

[CMCDragonkai]

The domainPath can be resolved by this change:

function domainPath(levels: DBDomain, key: string | ArrayBuffer): string | Buffer {
  // prefixer only takes string or Node Buffer
  // convert key into key_ which wraps ArrayBuffer as Buffer
  let key_: string | Buffer;
  if (typeof key === 'string') {
    key_ = key;
  } else if (ArrayBuffer.isView(key)) {
    key_ = Buffer.from(key.buffer, key.byteOffset, key.byteLength);
  } else {
    key_ = Buffer.from(key);
  }
  if (!levels.length) {
    return key_;
  }
  let prefix = key_;
  for (let i = levels.length - 1; i >= 0; i--) {
    prefix = prefixer(levels[i], prefix);
  }
  return prefix;
}

However this doesn't fix everything. We also use Buffer for this.crypto.key and there are complications with when using worker manager.

When using worker manager, we want to copy the ArrayBuffer, but if the actual instance is Node Buffer, the slice call creates a view, not a copy. So we have to distinguish by checking with Buffer.is and ArrayBuffer.isView and doing a set or copy or slice call. Ensuring that we are in fact copying the ArrayBuffer.

But yes this issue looks like it is totally possible.

[CMCDragonkai]

Note that this would only present an ArrayBuffer interface to the outside world. Internally Node Buffer would still be used by the domainPath. And of course LevelDB still returns Buffer. So primarily this change would be about increasing the compatibility of DB as we progressively move towards more usage of Uint8Array and ArrayBuffer such as js-id and webcrypto.

[CMCDragonkai]

The TextDecoder works on ArrayBuffer and Uint8Array.

[nix-shell:~/Projects/js-db]$ node
Welcome to Node.js v14.17.3.
Type ".help" for more information.
> u = new Uint8Array([0x61, 0x62, 0x63])
Uint8Array(3) [ 97, 98, 99 ]
> u.buffer
ArrayBuffer { [Uint8Contents]: <61 62 63>, byteLength: 3 }
> new TextDecoder().decode(u.buffer)
'abc'
> new TextDecoder().decode(u)
'abc'

[CMCDragonkai]

Need to ensure that this doesn't result in a performance regression with respect to benchmarks.