Retrying module import from other folders

[obones]

I have mono repo with a structure very similar to this:

Common
   |- Feature1
   |    \- index.js
   |- someUtils.js
   \- someOtherUtils.js
Lambda1
   |- index.js
   \- package.json
Lambda2
   |- index.js
   \- package.json
test
   |- Common
   |    |- Feature1
   |    |    |- index.js
   |    |    \- package.json
   |    |- package.json
   |    \- someUtils.js
   |- Lamda1
   |    |- index.js
   |    \- package.json
   \- Lamda2
        |- index.js
        \- package.json

The CI system creates the zip files that are deployed on the AWS Lambda system and does so by taking the content of lambda folder and placing next to it the content of the Common folder renamed as node_modules.
This has worked since the beginning and when unit tests were added later on, it was decided to have as little changes to the existing code as possible.
Each .js file inside the test folder uses module-alias to trick the loader into looking at the Common folder as a valid source for the files required by the lambda code. This allows debugging the unit tests while being able to edit the common files "in place", greatly speeding up the development process.
Here is the code added at the top of each .js file in the test subfolders:

const module_alias = require("module-alias");
module_alias.addPath(__dirname + "/../../Common");
module_alias.addPath(__dirname + "/node_modules/");

The second path alias is here because only the package.json files in the test folders are defining the devDependencies to @aws modules because they already are available in the target environment, and because the entry point is via tests only. This avoids cluttering the developer's machines with lots of duplicate folders.

This works fairly well in CommonJS format but as we must modernize the code base, we are trying to move to ES6 syntax and "module" resolution with as little changes as possible.

One thing that is clear is that adding path aliases like before is no longer possible because imports are resolved long before any of our code is run. This has to be done via the --import flag on the command line so that I can register a module hook which is what I tried with the following code, inspired by esm-module-alias:

import { register } from "node:module";
import { fileURLToPath } from "node:url";
import path from "node:path";

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

register("./loader.mjs", import.meta.url);

const windowsSupportString = process.platform === 'win32' ? 'file://' : '';
const base = removePrefix(__dirname, windowsSupportString); 

function removePrefix(value, prefix) {
   return value.startsWith(prefix) ? value.slice(prefix.length) : value;
}

function generateResolver() {
  return async (specifier, parentModuleURL, nextResolve) => {
    try
    {
        return await nextResolve(specifier, parentModuleURL);
    }
    catch (error)
    {
        if (error?.code === 'ERR_MODULE_NOT_FOUND')
        {
            let retryParentModuleURL = Object.assign({}, parentModuleURL);
            let retrySpecifier = specifier;

            retryParentModuleURL.parentURL = windowsSupportString + path.join(base, "..", "Common");

            try
            {
                console.log(`RETRY: ${retrySpecifier} from ${JSON.stringify(retryParentModuleURL)}`);
                return await nextResolve(retrySpecifier, retryParentModuleURL);
            }
            catch (innerError)
            {
                console.log("/INNER ------------");
                console.log(innerError);
                console.log(JSON.stringify({retrySpecifier, retryParentModuleURL}));
                console.log("/INNER ------------");
                throw innerError;
            }
        }

        throw error;
    }
  };
}

export const resolve = generateResolver();

This is not exactly the same as the situation before because the loader.mjs file is loaded once at the start of the process called by the test runner (mocha) which means __dirname is the one from that file, not the one for each test file.
But even then, the retry appears to work as I see the console log, but sadly it still throws an error, but this time with a hint message:

Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@aws-sdk/lib-dynamodb' imported from C:\Sources\myproject\src\api\test\Common
Did you mean to import "@aws-sdk/lib-dynamodb/dist-cjs/index.js"?

It's like it has "seen" the folder but did not find the expected files for ES modules, which I find weird as they appear to be in the dist-es folder.

Am I on the right track here? Would you have any suggestions for this situation?

I looked at subpath imports (with the # prefix) but this does not work because it cannot look into parent folders for resolution.

Thanks for your time.

[ljharb]

I'm confused - why not just take the actual project folder, run npm prune --production, and tar up everything that's left?

[obones]

Because that's a monorepo where each "Lambda" folder has its own pack script that creates a zip file. There must be one zip file per lambda as this is what's required by the AWS tools to deploy a lambda.

[ljharb]

ok, but you can still do that in each folder by running npm install --no-workspaces --production directly in each subfolder.

[obones]

But the main issue here is not really creating the deployment packages but rather have the IDE (VSCode with MochaTestExplorer) be able to find the dependencies and use them "in place" so that it's easier to debug and change the common files if need be.

[lizuju]

You are on the right general track with a loader, but I would change two details.

First, don't base the retry on the loader file's __dirname. A loader is initialized once, so that path tells you where the loader lives, not where the current import came from. The useful value is the resolve hook's context.parentURL. That is the module that is currently doing the import, and it is what lets you derive whether you are resolving from test/Lambda1, test/Lambda2, etc.

Second, parentURL should be a concrete file URL. In your retry you are effectively resolving from a directory-like URL:

C:\Sources\myproject\src\api\test\Common

That explains the error. Node is now trying to resolve @aws-sdk/lib-dynamodb as if it was imported from test/Common, so it searches from that location. It is no longer using the lambda test folder where your test-only node_modules exists.

For ESM I would split the two cases instead of trying to reproduce module-alias.addPath() exactly:

import { access } from 'node:fs/promises'
import path from 'node:path'
import { fileURLToPath, pathToFileURL } from 'node:url'

const commonDir = path.resolve(process.cwd(), 'test/Common')

async function exists(file) {
  try {
    await access(file)
    return true
  } catch {
    return false
  }
}

export async function resolve(specifier, context, nextResolve) {
  try {
    return await nextResolve(specifier, context)
  } catch (error) {
    if (error?.code !== 'ERR_MODULE_NOT_FOUND') throw error
  }

  // Only fallback your internal Common imports, not every package import.
  // Adjust this test to match the names you actually import from Common.
  if (!specifier.startsWith('.') && !specifier.startsWith('/') && !specifier.startsWith('node:')) {
    const commonIndex = path.join(commonDir, specifier, 'index.js')
    if (await exists(commonIndex)) {
      return {
        url: pathToFileURL(commonIndex).href,
        shortCircuit: true,
      }
    }

    const commonFile = path.join(commonDir, `${specifier}.js`)
    if (await exists(commonFile)) {
      return {
        url: pathToFileURL(commonFile).href,
        shortCircuit: true,
      }
    }
  }

  throw new Error(`Cannot resolve ${specifier} from ${context.parentURL}`)
}

The important part is that this fallback only maps the Common modules you own. For external packages like @aws-sdk/lib-dynamodb, I would let the normal resolver handle them from the original context.parentURL, which means the package needs to be resolvable from the lambda test package's node_modules or a parent node_modules.

If you really want to retry package resolution from another location, use a fake file inside that location, not the directory itself, for example:

await nextResolve(specifier, {
  ...context,
  parentURL: pathToFileURL(path.join(lambdaTestDir, '__resolve__.mjs')).href,
})

But I would keep that as a last resort. The cleaner long-term shape is still to make Common a real workspace/file dependency with explicit exports, because ESM resolution is much stricter than CommonJS' historical directory and NODE_PATH-style lookup behavior.