Migrate to ESM (and get dynamic await)

[CMCDragonkai]

What is your research hypothesis/question?

There is a conjunction of 4 features:

• dynamic imports
• top level await
• nodejs module type (es modules in nodejs)
• explicit exports in package.json

That enables some nifty capabilities:

• The ability to create modules that can do feature tests and import the right module for a given platform - cross platform compatibility (especially in terms of platform-native features vs importing specific modules)
• The ability to move away from commonjs modules and use esmodules as a standard for all platforms - this can reduce the dependence on bundling if all platforms support esmodules
• The ability to create explicit package exports, so one can create subdirectory imports import x from 'a/b/c'; - this can allow us to flexibly export things without having to go down the dist/... route all the time

These features are all coming online in various forms, however some initial experiments show that they involve alot of changes.

Review existing ideas, literature and prior work

1. motivation from nativescript React Native and Mobile OS (iOS and Android) Compatibility Polykey#155 (comment)
2. experiment in js-id ID standardisation for all projects and domains js-id#1 (comment)
3. ts-node support ESM support: soliciting feedback TypeStrong/ts-node#1007
4. Wait for TS 4.5 release

Research conclusion

See the notes in MatrixAI/js-logger#29

[CMCDragonkai]

It would be nice if changing to this didn't involve changing all of our import paths.

[CMCDragonkai]

Solving this will enable MatrixAI/js-id#2

[CMCDragonkai]

Pkg doesn't currently support ES modules. But it should soon: vercel/pkg#782.

Currently it may require us to use esbuild.

The esbuild could be trialed out to see if it's better than using webpack which we had removed before, and see if it can help also deal with non-transpiled files that are in ./src and automatically move to ./dist.

It may also help with doing some optimisation work like tree-shaking. But of course it doesn't make sense to put it all into one single file like we were doing in webpack. Library modules should be able to imported separately.

[CMCDragonkai]

Note that dynamic imports already work. Right now they are just translated to require calls. However being able to use these for cross-platform implementation swapping will require top level awaits.

[CMCDragonkai]

With #37 merged, we are one step closer.

Before working with ES modules and dynamic imports that use ES modules, it must be tested that it works in vercel pkg. Keep track of this issue: vercel/pkg#1603

Also important that it works with ts-node too.

[CMCDragonkai]

Experimental support for ESM in ts-node is now available: TypeStrong/ts-node#1007 (comment)

[CMCDragonkai]

Moving to ESM is a big task, there are other tools we need support for: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c