Package specific configuration (for apps and libraries)

[kraenhansen]

We've actively worked towards putting as few restrictions on the packages bringing Node-API modules as possible, to make it more likely to migrate / re-use existing packages intended for Node.js and we would like to continue to support packages providing Node-API modules without requiring package-specific configuration.

How to configure

We could rely on cosmiconfig (which is also used by the React Native Community CLI):

By default, Cosmiconfig will check the current directory for the following:

• a package.json property
• a JSON or YAML, extensionless "rc file"
• an "rc file" with the extensions .json, .yaml, .yml, .js, .ts, .mjs, or .cjs
• any of the above two inside a .config subdirectory
• a .config.js, .config.ts, .config.mjs, or .config.cjs file

We could load this config starting from the "app" and then look for it in all direct dependencies of the app package and all transitive dependencies explicitly enumerated - see below.

What to configure

Possible configuration options which could optimise the experience:

Library name mapping to paths

We could allow library authors to provide explicit paths for the Node-API prebuilds bundled with their package:

{
  // Alternative names for this property could be "addons" or "modules"? 🤔
  "prebuilds": [
    // Provide a path for a prebuild
    "./build/Release/somthing.node",
    // Which would be an alias for an object with the same path
    { "path": "./build/Release/somthing.node" },
    // Allow overriding the name (default would be "my-lib--addon") used for the library files when linked into the app
    { "name": "addon", "path": "./build/Release/addon.node" },
    // Allow specifying platform-specific paths
    {
      "name": "addon",
      "path": {
        "android": "./build/Release/addon.android.node",
        "ios": "./build/Debug/addon.apple.node"
      },
    },
  ]
}

Control scanning

We could group options related to scanning for Node-API modules, either under a "scanning" (or simply "scan") key.

We could enable scanning of specific transitive dependencies (see #367 for use-case)

{
  "scanning": {
    "dependencies": ["name-of-lib-with-node-api-module"]
  }
}

We could provide lists of glob patterns to include and exclude when scanning for modules

{
  "scanning": {
    "include": ["./build"],
    "exclude": ["./node_modules"],
  }
}

Or disable scanning all together:

{
  "scanning": {
    "disabled": true
  }
}

[owl352]

Yes, that sounds interesting. We could probably use package.json for both mapping and scan settings. We could introduce a nodeAPI property and describe everything that is needed within it. It might also be worth adding a check for package.json in dependencies from the root package.json, which would simplify the user flow for dependencies that use the node api of other dependencies.

example package.json

{
  "name": "app",
  "version": "1.0.0",
  "main": "index.ts",
  "scripts": {
    "start": "expo start",
  },
  "dependencies": {},
  "devDependencies": {},
  "nodeApi": {
    "scanning":  {
      "include": ["./module_path", "included_module_name"],
      "exclude": ["./directory", "module_name"],
    },
    "addons": [
      { "name": "addon", "path": "./build/Release" },
    ]
  }
  "private": true
}

If we also check package.json in dependencies from root package.json, the user will not have to manually write this property in the root package.json for their application. However, this will only work if the dependency developer includes this field in their package.json 😕

And perhaps for addons, it would be better to specify not the path to *.node, but to the directory that contains the addons. I just can't imagine a scenario where a user would need to specify individual files.

[kraenhansen]

Great! I had few thoughts based on that (likely too detailed, but it will serve well for my friend Claude).

You're right, I agree the file extension should probably be left out entirely when declaring the "path" of an addon 👍 And let's not allow specifying platform-specific paths for now - I agree that need is probably very niche.

Config property / file name

I don't think it'd be appropriate for this package to grab the nodeApi property in package.json. Perhaps in a future where it's more generalized, but for now I think we should prefix it with "react-native" too.

Since the host package is responsible for the linking, I'd reuse that name and allow the config to be stored in:

• reactNativeNodeApi key in package.json
• react-native-node-api.config.json
• react-native-node-api.config.js
• react-native-node-api.config.ts
• react-native-node-api.config.mjs
• react-native-node-api.config.cjs

Default behaviour

I think the defaults should match the behavior we already have.

App config

The default configuration for an app developer would enumerate all dependencies and peerDependencies for scanning as they could potentially contain Node-API modules and include all files of the app directory and exclude large known directories.

{
  "name": "my-app",
  "dependencies": {
    "my-lib": "^1.0.0"
  },
  "peerDependencies": {
    "another-lib": "^1.0.0"
  },
  "devDependencies": {
    "something-else": "^1.0.0"
  },
  "reactNativeNodeApi": {
    "scanning":  {
      "dependencies": ["my-lib", "another-lib"], // keys of all dependencies and peerDependencies
      "include": ["**"],
      "exclude": ["node_modules", ".git", "ios", "macos", "android"],
    }
  }
}

These defaults would be partially overwritable (i.e. if the app developer declare scanning.dependencies the default include and exclude would still be used.

Library configs

Every time a package is considered for scanning, it itself can declare a configuration, to control how it's scanned for Node-API modules and what transitive

For library packages (the dependencies from the app or transitively declared through

{
  "name": "my-lib",
  "dependencies": {
    "another-lib": "^1.0.0"
  },
  "reactNativeNodeApi": {
    "scanning":  {
      "dependencies": [], // No dependencies of libraries are scanned
      "include": ["**"],
      "exclude": ["node_modules", ".git"],
    }
  }
}

[owl352]

Overall, I like everything — it sounds logical 👍

I’d just like to clarify one thing: which configuration file format do we ultimately want to make the primary one? Or are we planning to support all options at once (json, js, ts, mjs, cjs + a key in package.json)?

I just want to understand whether there’s a priority format around which we’ll build the core logic, or if we’re going straight for full multi-format support.

In general, I can take on this task tomorrow if there are no objections 🙂

[kraenhansen]

That sounds great 💙 My initial thought would be to use https://github.com/cosmiconfig/cosmiconfig or perhaps https://github.com/antonk52/lilconfig to read the config object. Both libraries support reading from package.json, static JSON and JS/TS source-code.

Is that needed though? Well, it's a neat feature to be able to run code as part of building up the configuration object. We could however, definitely start out simply supporting a config in package.json via the dependencies we already use and add support for separate config files later as an additive change 👍

It would be awesome if you wanted to give this a go. It doesn't have to be all or nothing: I'm completely fine with merging and releasing just the feature of declaring transitive dependencies that you initially created a PR for 👍

Really happy to have had a chance to think a bit more about this 🙏 thanks

[shirakaba]

@kraenhansen:

App-level config

Regarding your proposed scanning config: I think it's a little confusing to have both dependencies living alongside include and exclude, but on the other hand, I can't think of any better way to offer the same level of flexibility (your proposal allows for scanning Node-API modules that don't live under node_modules). So, works well enough for me. 👍

I'm not wild about the long file name react-native-node-api.config.json, but on the other hand, it's more self-documenting than an acronym, so sure. 👍

Library-level config

// (@kraenhansen)
// Alternative names for this property could be "addons" or "modules"? 🤔
"prebuilds": [

I think we can improve on the name prebuilds, because the point is not the fact that the library may have been built in advance, but where to find the relevant build output. I'm half-inclined to suggest searchPaths, but as that makes me think of traditional linking, maybe binaries is better (as it makes it clear we're asking about the native build artifacts and not anything to do with JS).

(@owl352)
And perhaps for addons, it would be better to specify not the path to *.node, but to the directory that contains the addons. I just can't imagine a scenario where a user would need to specify individual files.

I'm a little confused here, as .node may refer to a file (for Node.js consumers) or a folder (for React Native consumers):

And I've forgotten whether the plan is for Apple-platform library authors to provide .xcframeworks or .frameworks. It's been a while.

But maybe that's all the more reason to just allow the user to point at the main build root and let our tooling just dig around and work out which are the relevant artifacts? 🤷‍♂️

Either way, for the sake of discussion, @kraenhansen could you illustrate the expected file structure of a cross-platform Node-API module (spanning React Native Android, React Native iOS, React Native macOS, and Node.js macOS) to help us pin down what would be the best design for how to specify paths to the relevant files/folders?

Both app-level and library-level configs

Happy to go with a cosmiconfig-like approach too, as it allows us to handle any sort of evolution with CJS and MJS. 👍

[kraenhansen]

Thansk for your comments @shirakaba!

confusing to have both dependencies living alongside include and exclude

Right - I can see how that could be a bit confusing 🤔

Basically, include and exclude would be used to control the file-system scanning for Node-API prebuilds "within" a certain package (thinking more about it, perhaps node_modules should always be unconfigurably excluded).

Whereas the dependencies would enumerate the names of all other packages that this particular package (library or app) would want to get scanned as well.

The include and exclude would not "transfer over" into the scanning of the other dependencies as these values would be specific to the package being scanned.

With the proposed defaults I wouldn't expect 90% of app developers to have to declare a configuration file, since all their direct dependencies would be scanned and those packages could themselves declare their (peer) dependencies to include them in the scanning.

Does that make more sense? 🙂 Or did you get that already and the words were mostly causing a potential confusion of others?

not wild about the long file name react-native-node-api.config.json

Yeah - me neither 🤔 It's an established pattern to name the config from the package name and it's also a proud tradition to have descriptive (and verbose) package names in the React Native ecosystem 🙈😆

I think we can improve on the name prebuilds

To clarify if it's not obvious, the way I see it, a library author would likely either

1. specify the scanning.include and scanning.exclude
2. disable scanning and instead provide an explicit list of these, optionally including their name to provide a smaller name (or outright disable the renaming we're doing to prevent collisions - that should be another option in that object BTW).

They could declare the files by both enumerating some and scanning for others, but that seem much less likely to me - perhaps we'd use that in some of the packages used for testing 🤔

Let's see:

• binaries: Something not text. Often (but not always native build artifacts). This is what they are, but less semantic (what kind and what are they supposed to be used for) than the other alternatives IMO.
• prebuilds: A type of binary which is build ahead-of-time (often by someone else). I agree using this word would be imprecise in the case of an app developer building these themselves as part of building their app. On the other hand, I've spoken about "structure of the prebuilds" and we even have a PREBUILDS.md document 🤔
• libraries / dynamicLibraries: More precise, but also - this is a place you go to pick up books. And the thing we call the "packages which are not apps" 😬
• modules / nativeModules: This is what the React Native community calls them and many places in the existing codebase these are referred to as "modules" - because I've always considered these an alternative to Nitro and Turbo Modules - but this also does clash with the idea of EcmaScript Modules.
• addons: This is what the Node.js project calls them.

It sort of feels like one of those things where we should pick one and go with it through the config, the docs, the codebase, etc.

Making the list above, I'm slightly leaning towards "addons", just because leans into prior-art and it seems to work for the Node.js project 🤷

I'm a little confused here, as .node may refer to a file (for Node.js consumers) or a folder (for React Native consumers)

To clarify my counter-suggestion after @owl352 pointed out the not to use *.node in the path, I'd amend the proposal above to leave out the file extension entirely. As per your example, this would be:

{ "path": "./build/RelWithDebInfo/NativeScript" }

Which would end up being linked in as "my-lib--NativeScript" (if the containing package name was "my-lib") and that final "library name"

I've forgotten whether the plan is for Apple-platform library authors to provide .xcframeworks or .frameworks

Xcframeworks containing sdk+arch specific slices (.framework) and actually, with #351 the important part is just that we get to the .frameworks. I think the important part is that they have to be to be signed in the final app bundle.

could you illustrate the expected file structure of a cross-platform Node-API module (spanning React Native Android, React Native iOS, React Native macOS, and Node.js macOS)

I could give that a go 👍 I agree it would help ground our common understanding - and it would be great for the docs page regarding prebuilds as well.

I'll see if I can get this sorted out in the coming days 🙏

[owl352]

Updated #367.

I'm not quite sure how to work with include and exclude yet. With include, it's unclear what name to assign to addons. Currently, the auto-linker uses the module name and the name before *.node.* (?) to form module_name--addon_name. If we specify a folder to search for addons in, we can specify: root project name, folder name, or use only addon_name. I believe that specifying the name of the folder with addons is not correct, as this could potentially create a name collision. I cannot fully evaluate the other options, as I have not studied the entire repository code. Perhaps for includes, it is worth changing the structure from an array of strings to an array of objects.

Like

"reactNativeNodeApi": {
    "scan": {
      "dependencies": ["dependencyName"],
      "include": [
        {
          "name": "addonName",
          "path": "./build"
        }
      ]
    }
  }

and as result we have addonName--addon_name

[kraenhansen]

I think you can leave include and exclude unimplemented for now. My intent with those would be that they served as the basis for the search happening somewhere around findNodeApiModulePaths:

react-native-node-api/packages/host/src/node/path-utils.ts

Line 363 in 80ae73b

excludePatterns = DEFAULT_EXCLUDE_PATTERNS,

But looking at this, we're not using globs and instead we're using regular expression matching on the normalized path. I would probably try to refactor that into using fs.promises.glob - but that's more involved 👍 You can just leave that to me as it's not on your path to success for getting "transitive dependencies" supported.

[shirakaba]

@kraenhansen Just a brief response to the bikeshedding above – I'll be happy with whatever you two land on 👍