-
Notifications
You must be signed in to change notification settings - Fork 698
chore(spanner-driver): sync spannerlib-node code #9025
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| # Node.js Spanner Wrapper Build and Release Architecture | ||
|
|
||
| This document describes the compilation pipeline, dual-publishing methodology, and native bridge linking process for the `spannerlib-node` driver. | ||
|
|
||
| ## The 3-Layer Pipeline | ||
|
|
||
| Bridging JavaScript to the native Go SDK involves a sequential compilation pipeline mapping V8 types to Go-compatible memory pointers across three separate layers: | ||
|
|
||
| ``` | ||
| [TypeScript Source] ---> (Babel/TSC) ---> [ESM & CJS JavaScript] | ||
| | | ||
| (node-gyp bridge) | ||
| v | ||
| [C++ Native Addon] | ||
| | | ||
| (cgo linker bridge) | ||
| v | ||
| [Go Shared Library] | ||
| ``` | ||
|
|
||
| ## Compilation Phases | ||
|
|
||
| ### Phase 1: Compiling the Go Shared Library (CGO Link) | ||
|
|
||
| Before building the Node.js Addon, the underlying Go codebase must be compiled into an object format that C/C++ can link against. | ||
| * **Trigger:** Executed via `npm run build:go` which runs `bash scripts/build-shared-lib.sh`. | ||
| * **Action:** The build script invokes the Go compiler with the `-buildmode=c-shared` flag, targeting the primary C-shared entry point located at [shared_lib.go](../../shared/shared_lib.go). | ||
| * **Outputs:** Generates a platform-specific native shared library (e.g., `libspannerlib.dylib` on macOS, `.so` on Linux, `.dll` on Windows) along with the corresponding C header file (`libspannerlib.h`). Both files are placed into the `spannerlib/shared/` directory. | ||
|
|
||
| ### Phase 2: Compiling the Native Bridge (node-gyp) | ||
|
|
||
| Once the Go shared library is generated, the Node.js C++ wrapper is compiled using `node-gyp` to map V8 engine objects into Go pointers. | ||
| * **Trigger:** Executed as part of `npm run build` which invokes `node-gyp rebuild`. | ||
| * **Action:** Reads the `gyp` build instructions in [binding.gyp](./binding.gyp) to locate the Go header files, and dynamically links the bridge against the generated Go shared object. It compiles the bridge source file [addon.cc](./src/cpp/addon.cc) using the local OS C++ compiler toolchain (e.g., Clang on macOS, GCC on Linux, MSVC on Windows). | ||
| * **Output:** Generates the native Node.js binary file at `build/Release/spanner_napi.node`. | ||
| * **Post-build Link Patch (macOS Only):** To ensure portability on macOS without requiring root or global library installs, `npm run postbuild` invokes the `install_name_tool`. This command alters the dynamic linker search path in the `.node` file to use `@loader_path/libspannerlib.dylib`, ensuring Node.js locates the Go dynamic library relatively from the compiled C++ bridge binary path. | ||
|
|
||
| ### Phase 3: TypeScript Compilation & Dual-Publishing (ESM / CJS) | ||
|
|
||
| Finally, the TypeScript layer (which handles user-facing API classes, JavaScript's `FinalizationRegistry` garbage collection mapping, and Protobuf serialization) is compiled for consumer consumption. The build is configured to output both **ES Modules (ESM)** and **CommonJS (CJS)** simultaneously, ensuring compatibility across modern and legacy Node.js environments. | ||
| * **Trigger:** Executed via `npm run compile`. | ||
| * **Action:** | ||
| 1. **ESM Setup:** `npm run compile:esm` runs the standard `tsc` (TypeScript Compiler) using the primary [tsconfig.json](./tsconfig.json) file. Outputs ES6 modules into `build/esm/`. | ||
| 2. **CJS Setup:** `npm run compile:cjs` compiles the codebase for legacy support using [tsconfig.cjs.json](./tsconfig.cjs.json) and pipes the JS output through `@babel/cli` to translate modern `import`/`export` keywords into standard CommonJS `require()` bindings. | ||
| 3. **Extension Fixer:** Because Node.js requires explicit file extensions when using ESM but Babel strips them or expects `.cjs` in CommonJS modes, the post-compilation script `node scripts/fix-extensions.cjs` rewrites the internal import paths across the `build/cjs/` folder to use `.cjs` extensions. | ||
| * **Output:** Generates two parallel directory trees under `build/esm` and `build/cjs`, making the distributed npm package natively dual-consumable using the `exports` configuration in `package.json`. | ||
|
|
||
| ## End-to-End Local Builds | ||
|
|
||
| To run the entire pipeline end-to-end and generate a fully runnable local build, invoke the top-level script: | ||
| ```bash | ||
| npm run build | ||
| ``` | ||
| This builds the underlying Go shared library, links the C++ bridge layer via `node-gyp`, patches dynamic linker paths, and outputs the final dual ESM/CJS JavaScript distributions. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| # Node-API Wrapper for Spanner Shared Library | ||
|
|
||
| This package provides a high-performance Node-API (N-API) bridge to the Go-based Spanner shared library. It offers superior stability and performance compared to traditional FFI approaches. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Node.js >= 20.0.0 | ||
| - Go compiler (to build the underlying shared library, if not pre-built) | ||
| - C++ toolchain (GCC/Clang or MSVC) | ||
|
|
||
| ## Installation & Building | ||
|
|
||
| To build the native addon and compile the driver for development, run: | ||
|
|
||
| ```bash | ||
| npm install | ||
| ``` | ||
|
|
||
| For a comprehensive architectural breakdown of the 3-layer compilation pipeline (Go Link, C++ node-gyp bridge, and Dual-publishing ESM/CJS), refer to the [BUILD_AND_RELEASE.md](file:///Users/gargsurbhi/Work/Github/go-sql-spanner/spannerlib/wrappers/spannerlib-node/BUILD_AND_RELEASE.md) documentation. | ||
|
|
||
| ## Usage | ||
|
|
||
| ```javascript | ||
| const { Pool, Connection } = require('spannerlib-node'); | ||
|
|
||
| async function run() { | ||
| const pool = await Pool.create(connectionString); | ||
|
|
||
| const conn = await pool.createConnection(); | ||
| const rows = await conn.executeSql('SELECT 1'); | ||
|
|
||
| while (await rows.next()) { | ||
| // process rows | ||
| } | ||
|
|
||
| await rows.close(); | ||
| await conn.close(); | ||
| await pool.close(); | ||
| } | ||
| ``` | ||
|
|
||
| ## Architecture | ||
|
|
||
| The wrapper consists of: | ||
| 1. **`src/cpp/addon.cc`**: C++ Node-API bridge that handles thread boundaries and type conversions between V8 and C. | ||
| 2. **`src/ffi/utils.ts`**: Helper functions to invoke native methods asynchronously using Promises. | ||
| 3. **`src/lib/`**: JavaScript classes (`Pool`, `Connection`, `Rows`) that provide a clean object-oriented interface. | ||
|
|
||
| ### Component Interaction | ||
|
|
||
| The wrapper operates by executing calls through the low-level native bridge (`addon.cc`). The user-facing classes (`Pool`, `Connection`, `Rows`) in the `lib` folder map execution requests into Protobuf payloads and pass them to the bridge via the asynchronous dispatcher in `utils.ts`. Each successful execution returns memory ID handles that are tracked inside the singleton state registry (`spannerlib.ts`) to link native garbage collection triggers. | ||
|
|
||
| ### Component Interaction & Memory Management | ||
|
|
||
| When a JavaScript object (like a `Pool` or `Connection`) is created, it holds an ID referencing a pinned Go object in memory. The `spannerLib` singleton maintains a **`FinalizationRegistry`**. This registry allows Node.js to listen for when the JavaScript object is garbage collected. When GC occurs, the registry automatically triggers a cleanup call to the native layer to release the corresponding Go object, preventing native memory leaks even if the developer forgets to call `.close()` explicitly. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| { | ||
| 'targets': [ | ||
| { | ||
| 'target_name': 'spanner_napi', | ||
| 'sources': [ 'src/cpp/addon.cc' ], | ||
| 'include_dirs': [ | ||
| '<!@(node -p "require(\'node-addon-api\').include")', | ||
| './shared', | ||
| ], | ||
| 'dependencies': [ | ||
| '<!(node -p "require(\'node-addon-api\').gyp")' | ||
| ], | ||
| 'cflags!': [ '-fno-exceptions' ], | ||
| 'cflags_cc!': [ '-fno-exceptions' ], | ||
| 'xcode_settings': { | ||
| 'MACOSX_DEPLOYMENT_TARGET': '10.15', | ||
| 'GCC_ENABLE_CPP_EXCEPTIONS': 'YES', | ||
| 'CLANG_CXX_LIBRARY': 'libc++', | ||
| }, | ||
| 'msvs_settings': { | ||
| 'VCCLCompilerTool': { 'ExceptionHandling': 1 }, | ||
| }, | ||
| 'conditions': [ | ||
| ['OS=="mac"', { | ||
| 'libraries': [ | ||
| '<(module_root_dir)/./shared/libspanner.dylib' | ||
| ], | ||
| 'xcode_settings': { | ||
| 'OTHER_LDFLAGS': [ | ||
| '-Wl,-rpath,@loader_path' | ||
| ] | ||
| }, | ||
| 'copies': [ | ||
| { | ||
| 'destination': '<(PRODUCT_DIR)', | ||
| 'files': [ '<(module_root_dir)/./shared/libspanner.dylib' ] | ||
| } | ||
| ] | ||
| }], | ||
| ['OS=="linux"', { | ||
| 'ldflags': [ | ||
| '-Wl,-rpath,$$ORIGIN' | ||
| ], | ||
| 'libraries': [ | ||
| '<(module_root_dir)/./shared/libspanner.so' | ||
| ], | ||
| 'copies': [ | ||
| { | ||
| 'destination': '<(PRODUCT_DIR)', | ||
| 'files': [ '<(module_root_dir)/./shared/libspanner.so' ] | ||
| } | ||
| ] | ||
| }], | ||
| ['OS=="win"', { | ||
| 'libraries': [ | ||
| '<(module_root_dir)/./shared/libspanner.dll' | ||
| ], | ||
| 'copies': [ | ||
| { | ||
| 'destination': '<(PRODUCT_DIR)', | ||
| 'files': [ '<(module_root_dir)/./shared/libspanner.dll' ] | ||
| } | ||
| ] | ||
| }] | ||
| ] | ||
| } | ||
| ] | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| let customConfig = []; | ||
| let hasIgnoresFile = false; | ||
| try { | ||
| require.resolve('./eslint.ignores.cjs'); | ||
| hasIgnoresFile = true; | ||
| } catch { | ||
| // eslint.ignores.js doesn't exist | ||
| } | ||
|
|
||
| if (hasIgnoresFile) { | ||
| const ignores = require('./eslint.ignores.cjs'); | ||
| customConfig = [{ignores}]; | ||
| } | ||
|
|
||
| module.exports = [...customConfig, ...require('gts')]; |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| module.exports = ['build/'] |
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,64 @@ | ||||||||||||||||||||||||||||||||||||||
| { | ||||||||||||||||||||||||||||||||||||||
| "name": "spannerlib-node", | ||||||||||||||||||||||||||||||||||||||
| "version": "0.1.0", | ||||||||||||||||||||||||||||||||||||||
| "main": "./build/cjs/src/index.cjs", | ||||||||||||||||||||||||||||||||||||||
| "module": "./build/esm/src/index.js", | ||||||||||||||||||||||||||||||||||||||
| "types": "./build/cjs/src/index.d.ts", | ||||||||||||||||||||||||||||||||||||||
| "type": "module", | ||||||||||||||||||||||||||||||||||||||
| "exports": { | ||||||||||||||||||||||||||||||||||||||
| ".": { | ||||||||||||||||||||||||||||||||||||||
| "import": { | ||||||||||||||||||||||||||||||||||||||
| "types": "./build/esm/src/index.d.ts", | ||||||||||||||||||||||||||||||||||||||
| "default": "./build/esm/src/index.js" | ||||||||||||||||||||||||||||||||||||||
| }, | ||||||||||||||||||||||||||||||||||||||
| "require": { | ||||||||||||||||||||||||||||||||||||||
| "types": "./build/cjs/src/index.d.ts", | ||||||||||||||||||||||||||||||||||||||
| "default": "./build/cjs/src/index.cjs" | ||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||
| }, | ||||||||||||||||||||||||||||||||||||||
| "engines": { | ||||||||||||||||||||||||||||||||||||||
| "node": ">=20.0.0" | ||||||||||||||||||||||||||||||||||||||
| }, | ||||||||||||||||||||||||||||||||||||||
| "scripts": { | ||||||||||||||||||||||||||||||||||||||
| "build:go": "bash scripts/build-shared-lib.sh", | ||||||||||||||||||||||||||||||||||||||
| "build": "npm run build:go && node-gyp rebuild && npm run compile", | ||||||||||||||||||||||||||||||||||||||
| "postbuild": "node -e \"if (process.platform === 'darwin') require('child_process').execSync('install_name_tool -change libspanner.dylib @loader_path/libspanner.dylib ./build/Release/spanner_napi.node')\"", | ||||||||||||||||||||||||||||||||||||||
| "compile:esm": "tsc -p .", | ||||||||||||||||||||||||||||||||||||||
| "compile:cjs": "tsc -p ./tsconfig.cjs.json && babel build/cjs --out-dir build/cjs --out-file-extension .cjs && node scripts/fix-extensions.cjs", | ||||||||||||||||||||||||||||||||||||||
| "compile": "npm run compile:esm && npm run compile:cjs", | ||||||||||||||||||||||||||||||||||||||
| "test:esm": "mocha build/esm/test/**/*.js", | ||||||||||||||||||||||||||||||||||||||
| "test:cjs": "mocha build/cjs/test/**/*.cjs", | ||||||||||||||||||||||||||||||||||||||
| "test": "npm run test:esm && npm run test:cjs", | ||||||||||||||||||||||||||||||||||||||
| "lint": "gts lint", | ||||||||||||||||||||||||||||||||||||||
| "clean": "gts clean", | ||||||||||||||||||||||||||||||||||||||
| "fix": "gts fix", | ||||||||||||||||||||||||||||||||||||||
| "prepare": "npm run compile", | ||||||||||||||||||||||||||||||||||||||
| "pretest": "npm run compile" | ||||||||||||||||||||||||||||||||||||||
| }, | ||||||||||||||||||||||||||||||||||||||
| "files": [ | ||||||||||||||||||||||||||||||||||||||
| "build/esm/src", | ||||||||||||||||||||||||||||||||||||||
| "build/cjs/src", | ||||||||||||||||||||||||||||||||||||||
| "build/Release/*.node" | ||||||||||||||||||||||||||||||||||||||
| ], | ||||||||||||||||||||||||||||||||||||||
| "dependencies": { | ||||||||||||||||||||||||||||||||||||||
| "node-addon-api": "^8.0.0", | ||||||||||||||||||||||||||||||||||||||
| "bindings": "^1.5.0", | ||||||||||||||||||||||||||||||||||||||
| "@google-cloud/spanner": "^8.7.1" | ||||||||||||||||||||||||||||||||||||||
| }, | ||||||||||||||||||||||||||||||||||||||
| "devDependencies": { | ||||||||||||||||||||||||||||||||||||||
| "mocha": "^11.0.1", | ||||||||||||||||||||||||||||||||||||||
| "typescript": "^6.0.0", | ||||||||||||||||||||||||||||||||||||||
| "@types/node": "^24.0.0", | ||||||||||||||||||||||||||||||||||||||
| "@types/bindings": "^1.5.0", | ||||||||||||||||||||||||||||||||||||||
| "@types/mocha": "^10.0.6", | ||||||||||||||||||||||||||||||||||||||
| "@babel/core": "^8.0.0", | ||||||||||||||||||||||||||||||||||||||
| "@babel/cli": "^8.0.0", | ||||||||||||||||||||||||||||||||||||||
| "sinon": "^22.0.0", | ||||||||||||||||||||||||||||||||||||||
| "@types/sinon": "^21.0.0", | ||||||||||||||||||||||||||||||||||||||
| "gts": "^7.0.0", | ||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+51
to
+59
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Several devDependencies specify non-existent or hallucinated major versions (e.g.,
Suggested change
References
|
||||||||||||||||||||||||||||||||||||||
| "@grpc/grpc-js": "^1.12.0", | ||||||||||||||||||||||||||||||||||||||
| "@grpc/proto-loader": "^0.8.0" | ||||||||||||||||||||||||||||||||||||||
| }, | ||||||||||||||||||||||||||||||||||||||
| "gypfile": false | ||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| #!/bin/bash | ||
| set -e | ||
|
|
||
| # Copyright 2026 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # Builds the shared library and places it in the shared directory. | ||
| # This script handles OS detection to use the correct file extension. | ||
|
|
||
| log() { | ||
| echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" | ||
| } | ||
|
|
||
| log "Starting Spannerlib Shared Library Build for Node Wrapper..." | ||
|
|
||
| # Resolve absolute paths | ||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| SHARED_LIB_DIR="$(cd "$SCRIPT_DIR/../shared" && pwd)" | ||
|
|
||
| log "Script Directory: $SCRIPT_DIR" | ||
| log "Shared Lib Directory: $SHARED_LIB_DIR" | ||
|
|
||
| # Auto-detect OS | ||
| case "$(uname -s)" in | ||
| Linux*) OS="Linux";; | ||
| Darwin*) OS="macOS";; | ||
| CYGWIN*|MINGW*|MSYS*) OS="Windows";; | ||
| *) OS="Unknown";; | ||
| esac | ||
| log "Auto-detected OS: $OS" | ||
|
|
||
| if [ "$OS" == "macOS" ]; then | ||
| echo "Building for macOS..." | ||
| go build -C "$SHARED_LIB_DIR" -o libspanner.dylib -buildmode=c-shared shared_lib.go | ||
| elif [ "$OS" == "Linux" ]; then | ||
| echo "Building for Linux..." | ||
| go build -C "$SHARED_LIB_DIR" -o libspanner.so -buildmode=c-shared shared_lib.go | ||
| elif [ "$OS" == "Windows" ]; then | ||
| echo "Building for Windows..." | ||
| go build -C "$SHARED_LIB_DIR" -o libspanner.dll -buildmode=c-shared shared_lib.go | ||
| else | ||
| echo "Unsupported operating system: $OS" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "Build complete." |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| // Copyright 2026 Google LLC | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| const fs = require('fs'); | ||
| const path = require('path'); | ||
|
|
||
| const cjsDir = path.join(__dirname, '../build/cjs'); | ||
|
|
||
| function traverseDir(dir) { | ||
| fs.readdirSync(dir).forEach(file => { | ||
| const fullPath = path.join(dir, file); | ||
| if (fs.statSync(fullPath).isDirectory()) { | ||
| traverseDir(fullPath); | ||
| } else if (fullPath.endsWith('.cjs')) { | ||
| let content = fs.readFileSync(fullPath, 'utf8'); | ||
| // Replace require paths starting with . / or .. / and ending with .js | ||
| content = content.replace(/require\(['"]((\.|\.\.)\/[^'"]+)\.js['"]\)/g, "require('$1.cjs')"); | ||
| // Fix import.meta.url syntax error in CommonJS | ||
| content = content.replace(/import\.meta\.url/g, '""'); | ||
| fs.writeFileSync(fullPath, content); | ||
| } else if (fullPath.endsWith('.js')) { | ||
| // Delete the original .js file generated by tsc | ||
| fs.unlinkSync(fullPath); | ||
| } | ||
| }); | ||
| } | ||
|
|
||
| if (fs.existsSync(cjsDir)) { | ||
| traverseDir(cjsDir); | ||
| console.log('Fixed extensions in .cjs files and removed .js files in build/cjs'); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The link to
BUILD_AND_RELEASE.mduses a hardcoded local absolutefile:///URL pointing to the author's local directory. This link will be broken for anyone else reading the documentation. Please change it to a relative link.