diff --git a/core/CODEMETA.md b/core/CODEMETA.md new file mode 100644 index 000000000000..d6208244c636 --- /dev/null +++ b/core/CODEMETA.md @@ -0,0 +1,253 @@ +# Google Cloud Node.js Core Architectural Guide +# *WARNING*: This file is AI generated and may contain inaccuracies. + +Welcome to the core architecture description for the `google-cloud-node` ecosystem. The packages under the `/core` directory form the complete shared base layer, build-time toolchain, request middleware, authentication flow, and network transport systems that power all downstream Google Cloud Platform client libraries (e.g., `@google-cloud/storage`, `@google-cloud/pubsub`, `@google-cloud/bigquery`, etc.). + +--- + +## πŸ—οΈ Core Layer Architecture + +The diagram below shows the structural divisions and dependencies among the core packages. They span from build-time code generation to runtime request execution, protocol binding, authentication, and logging. + +```mermaid +graph TD + %% Color and Style Definitions + classDef build fill:#f9f,stroke:#333,stroke-width:2px; + classDef high fill:#bbf,stroke:#333,stroke-width:2px; + classDef mid fill:#ddf,stroke:#333,stroke-width:2px; + classDef auth fill:#ffd,stroke:#333,stroke-width:2px; + classDef transport fill:#dfd,stroke:#333,stroke-width:2px; + classDef log fill:#fdd,stroke:#333,stroke-width:2px; + + subgraph "Build-Time & Generation Layer" + A["gapic-generator-typescript"]:::build + B["gapic-tools (tools)"]:::build + end + + subgraph "High-Level Client Framework Layer" + C["@google-cloud/common (common)"]:::high + D["@google-cloud/paginator (paginator)"]:::high + end + + subgraph "Request Orchestration & Protocol Binding (Middleware)" + E["google-gax (gax)"]:::mid + F["nodejs-googleapis-common"]:::mid + end + + subgraph "Primary Binary Transport (gRPC)" + GRPC["@grpc/grpc-js"]:::transport + end + + subgraph "Environment & Authentication Layer" + G["google-auth-library-nodejs"]:::auth + H["@google-cloud/gcp-metadata (gcp-metadata)"]:::auth + end + + subgraph "Low-Level HTTP & Fallback Transport" + I["gaxios"]:::transport + J["@google-cloud/teeny-request (teeny-request)"]:::transport + K["@google-cloud/retry-request (retry-request)"]:::transport + end + + subgraph "Diagnostic Logging" + L["google-logging-utils (logging-utils)"]:::log + end + + %% Dependencies / Relationships + A -->|Generates clients using| E + B -->|Transpiles & optimizes protos/mocking for| A + C -->|Wraps requests & authenticates with| G + D -->|Auto-paginates streams & callbacks for| C + E -->|Orchestrates requests for generated clients| F + E -->|Primary high-performance transport via| GRPC + E -->|HTTP/1.1 REST Fallback transport via| I + F -->|Dynamic API binding / REST fallback via| I + G -->|Queries GCP environment credentials from| H + G -->|Dispatches HTTP exchanges via| I + H -->|IP/DNS connection racing via| I + J -->|Fetch wrapper mimicking request API| I + K -->|Request retry wrapper, deprecated| J + L -.->|Zero-overhead diagnostic logging across| C + L -.->|Zero-overhead diagnostic logging across| E +``` + +--- + +## πŸ“‚ Package Registry & Core Responsibilities + +Below is a summary of each core package, explaining what it is, what it does, and how it fits as a building block in the ecosystem. + +### 1. [google-gax](packages/gax/CODEMETA.md) (`google-gax`) +* **Role**: Core Request Orchestration & Protocol Middleware. +* **Purpose**: Bridges high-level generated client libraries (GAPIC) with low-level transport layersβ€”primarily high-performance binary gRPC (`@grpc/grpc-js`) and HTTP/1.1 JSON-REST Fallback (`gaxios`). Since gRPC is the standard protocol for almost all high-throughput Google Cloud services (like Pub/Sub, Firestore, Bigtable, and Spanner), GAX acts as the primary wrapper around `@grpc/grpc-js` to orchestrate channel lifecycles and intercept RPC calls. +* **Key Mechanisms**: + * **gRPC Channel & Client Lifecycle**: Loads static `.proto` files or JSON descriptors, manages channel connections, handles mutual TLS (mTLS) handshakes, and configures standard client settings. + * `createApiCall`: Central factory that intercepts gRPC call invocations to apply timeouts, deadline schedules, retry backoffs, and descriptor behaviors. + * **APICallers & Descriptors**: Wraps low-level `@grpc/grpc-js` calls into specialized executors: Unary (retries + backoff), Pagination (async-iterables/collection unrolling), Bundling (batching concurrent client requests to reduce network overhead), Streaming (wraps raw gRPC streams using `duplexify` for resilient duplex proxying and stream resumption), and Long-Running Operations (polling gRPC operation states). + * **REST Fallback & Transcoding**: Directs browser or HTTP fallback environments by converting Protobuf declarations into valid JSON REST payloads, matching URL routes, and chunking continuous streaming array streams. + +### 2. [google-auth-library-nodejs](packages/google-auth-library-nodejs/CODEMETA.md) (`google-auth-library`) +* **Role**: Shared Authentication & Credential Engine. +* **Purpose**: Resolves credentials and manages OAuth 2.0 token loops, Service Accounts assertions, and federated identity tokens exchanges. +* **Key Mechanisms**: + * **Application Default Credentials (ADC)**: Sequentially probes environments (`GOOGLE_APPLICATION_CREDENTIALS`), well-known local gcloud configuration paths, and local metadata servers to instantiate appropriate auth clients. + * **Credential Subclasses**: Implements specific behaviors for JSON Web Tokens (`JWT`), standard OAuth (`OAuth2Client`), Cloud Metadata Server (`Compute`), and Security Token Service (`stscredentials`). + * **Workload Identity Federation**: Manages token exchange (AWS, Azure, Kubernetes, or pluggable scripts) via RFC 8693 token endpoints, supporting downstream Service Account impersonation. + * **Dynamic Crypto Factory**: Bridges Node.js environment (using native C++ `crypto` bindings) with browser runtimes (using standard Web Crypto). + +### 3. [gcp-metadata](packages/gcp-metadata/CODEMETA.md) (`gcp-metadata`) +* **Role**: Environment Detection & Metadata Server Client. +* **Purpose**: Queries the GCP Metadata Server (vital for serverless, Compute Engine, and GKE workloads) to fetch project IDs, regional zones, and service account OAuth tokens. +* **Key Mechanisms**: + * **Residency Verification**: Probes Linux bios/DMI vendors (`/sys/class/dmi/id/bios_vendor`), network MAC addresses (checks matching `42:01` prefixes), and serverless environment variables to detect GCP hosting. + * **Connection Racing**: Speeds up checks by racing an IP-based query (`169.254.169.254`) against a DNS-based query (`metadata.google.internal.`) using `Promise.any` to avoid DNS resolution lags. + * **Safe Parsing**: Employs `json-bigint` to parse large server ID numbers without loss of precision. + +### 4. [nodejs-googleapis-common](packages/nodejs-googleapis-common/CODEMETA.md) (`googleapis-common`) +* **Role**: Dynamic Discovery & API Orchestrator (Internal Helper). +* **Purpose**: Serves as the dynamic engine room for generated API clients by mapping Google Discovery Documents to executable JavaScript namespaces. +* **Key Mechanisms**: + * `createAPIRequest`: The operational center that deep-merges options, maps aliased parameters, resolves path variables via URL template expansion, serializes arrays, and tracks multipart stream uploads. + * **Trusted Partner Cloud (TPC)**: Intercepts and routes target URLs to secure partner domains based on configurations, cross-validating credentials to prevent leakage. + * **HTTP/2 Session Pooling**: Pools multiplexed HTTP/2 sessions with proactive inactive session teardowns. + +### 5. [gaxios](packages/gaxios/CODEMETA.md) (`gaxios`) +* **Role**: Resilient HTTP/Fetch Client. +* **Purpose**: Provides an `axios`-like interface built on top of native `fetch` (with a fallback to `node-fetch`), optimized for Google Cloud services. +* **Key Mechanisms**: + * **Exponential Backoff Retries**: Configurable backoff multiplier and randomized jitter to reschedule requests during server failures. + * **Mutual TLS (mTLS)**: Resolves mTLS client certificates for secure HTTPS network channels. + * **Interceptors**: Supports asynchronous request and response pipelines. + * **Log Redaction**: Protects against credential leaks by automatically sanitizing authorization headers and URL search params from logs and errors. + +### 6. [teeny-request](packages/teeny-request/CODEMETA.md) (`teeny-request`) +* **Role**: Lightweight Fetch-based Request Wrapper. +* **Purpose**: A minimal wrapper around `node-fetch` designed to replicate the legacy `request` API signature, minimizing bundle weight while ensuring compatibility. +* **Key Mechanisms**: + * **Option Normalization**: Converts legacy options into standard `RequestInit` configurations. + * **Concurrency Management (`TeenyStatistics`)**: Tracks concurrent outbound requests. If counts cross warning thresholds (default: 5000), it logs warning notices once via `process.emitWarning` to prevent thread/socket exhaustion. + * **Proxy Filtering**: Handles `NO_PROXY` regular expressions and wildcard bypass mappings. + +### 7. [common](common/CODEMETA.md) (`@google-cloud/common`) +* **Role**: Base Inheritance & Resource Modeling Framework. +* **Purpose**: Establishes the abstract class structures and request/retry pipelines for all hand-written Google Cloud client libraries. +* **Key Mechanisms**: + * `Service`: Models a service endpoint (e.g., `storage.googleapis.com`). Coordinates authentication, intercepts requests, extracts active Project IDs, and injects user telemetry headers. + * `ServiceObject`: Models a specific cloud entity (like a Bucket, Dataset, or Pub/Sub Topic). Provides generic CRUD operations (`create`, `get`, `delete`, `setMetadata`) and forwards requests to the parent service, appending its own ID to paths. + * `Operation`: Extends `ServiceObject` to implement exponential polling loops for Long-Running Operations (LRO). + +### 8. [paginator](paginator/CODEMETA.md) (`@google-cloud/paginator`) +* **Role**: Callback, Promise, & Streaming Pagination Wrapper. +* **Purpose**: Wraps paginated, cursor-based client methods to simplify result collection. +* **Key Mechanisms**: + * `extend`: Monkey-patches class prototypes to wrap target methods. + * **Auto-Pagination**: Transparently fetches all pages of a resource and compiles results, returning them via a single Callback or resolved Promise. + * **Transform Stream (`ResourceStream`)**: Exposes an object-mode stream that fetches pages on-demand as downstream readers consume data, gracefully managing backpressure and API limits. + +### 9. [logging-utils](packages/logging-utils/CODEMETA.md) (`google-logging-utils`) +* **Role**: Zero-Overhead Diagnostic Logging Utility. +* **Purpose**: Provides an optimized diagnostic logging layer that remains completely silent in production and is highly descriptive when enabled. +* **Key Mechanisms**: + * **Zero-Overhead Fallback**: If logging is not enabled via `GOOGLE_SDK_NODE_LOGGING`, calls immediately return a static no-op placeholder function, avoiding allocations. + * **Pluggable Backends**: Swaps globally between colorized text console logging (`NodeBackend`), existing application logger hooks (`DebugBackend`), and GCP-compliant structured JSON logging (`StructuredBackend`). + * **Event Emitter Hooks**: Every namespaced logger acts as an `EventEmitter` emitting `'log'` packets, letting parent libraries tap into diagnostics. + +### 10. [gapic-generator-typescript](generator/gapic-generator-typescript/CODEMETA.md) +* **Role**: Client Libraries Compiler Pipeline. +* **Purpose**: A `protoc` plugin and CLI compiler that parses Protocol Buffer (`.proto`) files into high-quality, production-ready TypeScript/JavaScript clients. +* **Key Mechanisms**: + * **Metadata Augmentation**: Translates proto HTTP, pagination, retry, batching, and long-running operation annotations into structured configurations. + * **Nunjucks Template Engine**: Utilizes templates to render all output source code (clients, index files, documentation, configurations, and mocha/sinon test suites). + * **Verification Baselines**: Compares compiler outputs line-by-line against checked-in reference baselines (`baselines/` and `baselines-esm/`) to prevent regressions. + +### 11. [tools](packages/tools/CODEMETA.md) (`gapic-tools`) +* **Role**: Build-Time Proto Compilers & Babel Transformations. +* **Purpose**: Provides the build scripting and Babel AST plugins needed to package, compile, and optimize client libraries for dual CJS/ESM distribution. +* **Key Mechanisms**: + * `compileProtos`: Executes `pbjs`/`pbts` to generate JS/TS files, relaxing union types (Enums, Bytes, Longs) to provide developers with flexible API parameters. + * `prepublishProtos`: Gathers and copies Google's shared common proto files into local package folders so packages are self-contained before NPM publishing. + * **Babel AST Plugins**: Swaps ESM expressions (`import.meta.url`) for CJS equivalents (`__dirname`), toggles compilation flags, and rewrites test mocking imports (`esmock` βž” `proxyquire`). + +### 12. [retry-request](packages/retry-request/CODEMETA.md) (`@google-cloud/retry-request`) +* **Role**: Request Retry Wrapper (*Deprecated*). +* **Purpose**: A legacy HTTP request wrapper that automatically retried transient transport and HTTP errors using exponential backoff and jitter. +* **Note**: Deprecated as of July 2024. Active HTTP retry and backoff capabilities have been consolidated directly inside the [gaxios](#5-gaxios-gaxios) client library. + +--- + +## πŸ”„ Runtime Execution Lifecycle + +The sequence diagram below illustrates how code execution flows through these building blocks. It tracks how a high-level application call propagates down through middleware, environment detection, authentication, and transport layers before returning a response. + +```mermaid +sequenceDiagram + autonumber + actor App as Application Code + participant Client as Service Client (Generated GAPIC / Common-based) + participant GAX as google-gax (Middleware) + participant Auth as google-auth-library (Auth Engine) + participant Meta as gcp-metadata (Metadata Server) + participant GRPC as @grpc/grpc-js (Primary Transport) + participant GaxCl as Gaxios / Fetch (Fallback Transport) + participant Serv as Google Cloud API Endpoint + + rect rgb(240, 245, 255) + Note over App, Client: 1. API Invocation + App->>Client: client.listInstances(options) + end + + rect rgb(235, 255, 235) + Note over Client, GAX: 2. Request Orchestration & Interception + alt Client is generated GAPIC library + Client->>GAX: apiCall(requestObject) + GAX->>GAX: Apply Bundling / Pagination / Timeout Policies + else Client is hand-written (Common-based) + Client->>Client: ServiceObject.request() / Paginator.run_() + end + end + + rect rgb(255, 255, 240) + Note over GAX, Meta: 3. Authentication & Environment Discovery + GAX->>Auth: getClient() / getAccessToken() + activate Auth + Auth->>Auth: Check Application Default Credentials (ADC) + alt Credentials File exists + Auth->>Auth: Load JWT Private Key / Workload config + else Running on Google Cloud Server + Auth->>Meta: isAvailable() / instance/service-accounts/default/token + activate Meta + Meta->>Meta: IP/DNS Connection Racing (169.254.169.254 vs metadata.google.internal.) + Meta-->>Auth: Return Access Token + deactivate Meta + end + Auth-->>GAX: Return Authenticated Client / Bearer Token + deactivate Auth + end + + rect rgb(240, 255, 240) + Note over GAX, Serv: 4. Request Transport & Execution + alt Transport is native gRPC (Default / Primary) + GAX->>GRPC: Invoke RPC method on gRPC client + activate GRPC + GRPC->>Serv: Binary HTTP/2 RPC Request with Metadata + alt Transient gRPC Failure (e.g., UNAVAILABLE) + GRPC-->>GAX: Return status code (e.g., 14) + GAX->>GAX: Calculate Backoff Delay (retries.ts) + GAX->>GRPC: Retry RPC call + end + GRPC-->>GAX: Return Protobuf response object / stream + deactivate GRPC + else Transport is HTTP/1.1 JSON-REST Fallback + GAX->>GaxCl: gaxios.request(transcodedOptions) + activate GaxCl + GaxCl->>Serv: HTTP JSON Fetch Request with Auth Header + alt Transient HTTP failure (e.g., 429 / 5xx) + GaxCl->>GaxCl: Calculate Exponential Backoff + Jitter + GaxCl->>Serv: Replay request + end + GaxCl-->>GAX: Return REST JSON Response + deactivate GaxCl + end + Serv-->>App: Return final structured API response / stream + end +``` diff --git a/core/common/CODEMETA.md b/core/common/CODEMETA.md new file mode 100644 index 000000000000..7dcf87885ce9 --- /dev/null +++ b/core/common/CODEMETA.md @@ -0,0 +1,122 @@ +# `@google-cloud/common` Codebase Metadata +# *WARNING*: This file is AI generated and may contain inaccuracies. + +This document provides an architectural overview, file map, and development reference for the `@google-cloud/common` package located in the `core/common` directory. + +--- + +## πŸ“– Overview + +`@google-cloud/common` is the core foundational layer for Google Cloud Platform (GCP) Node.js client libraries. It standardizes HTTP request construction, authentication, retries, stream wrapping, and resource model representations. Rather than having every service library (e.g., `@google-cloud/storage`, `@google-cloud/bigquery`, or `@google-cloud/pubsub`) implement its own authentication, request building, and error handling, they inherit from or compose the classes and utilities provided by this package. + +### Core Design Philosophy +1. **Inheritance-based Architecture**: High-level services and resources inherit from abstract base classes (`Service` and `ServiceObject`) to share behavior like authentication and generic CRUD methods. +2. **Unified Request Lifecycle**: All REST API requests run through a single pipeline that automatically handles token updates, endpoint overrides, request interception, custom user-agents, and retry policies. +3. **Callback-to-Promise Promisification**: Standard methods are defined as callback-based internally, and automatically wrapped with `@google-cloud/promisify` to expose an elegant Promise/async-await API to end-users. + +--- + +## πŸ“ Core Architecture & Hierarchy + +The interaction between the primary classes and utilities can be visualized as follows: + +```mermaid +classDiagram + class Service { + +String baseUrl + +String apiEndpoint + +GoogleAuth authClient + +Interceptor[] interceptors + +request(reqOpts, callback) + +requestStream(reqOpts) + +getProjectId() + } + + class ServiceObject { + +ServiceObjectParent parent + +String id + +Object metadata + +create(options, callback) + +delete(options, callback) + +get(options, callback) + +getMetadata(options, callback) + +setMetadata(metadata, options, callback) + +request(reqOpts, callback) + } + + class Operation { + +Int completeListeners + +Boolean hasActiveListeners + +promise() + #startPolling_() + #poll_(callback) + } + + class util { + +ApiError + +PartialFailureError + +makeAuthenticatedRequestFactory(config) + +makeRequest(reqOpts, config, callback) + +decorateRequest(reqOpts, projectId) + +shouldRetryRequest(err) + } + + ServiceObject <|-- Operation : Inherits + ServiceObject --> ServiceObjectParent : Has Parent (typically Service) + Service --> util : Uses for requests & auth + ServiceObject --> util : Uses for options parsing +``` + +--- + +## πŸ—‚οΈ Source Code File Map (`src/`) + +The primary logic of the package is isolated in the `src/` directory: + +| File Name | Primary Exports | Detailed Responsibility | +| :--- | :--- | :--- | +| **[`index.ts`](src/index.ts)** | `Service`, `ServiceObject`, `Operation`, `util`, `ApiError`, `AbortableDuplex` | The main entry point of the package. It acts as a barrel export, aggregating and re-exporting all public classes, types, and utility collections for downstream client libraries to consume. | +| **[`service.ts`](src/service.ts)** | `Service`, `ServiceConfig`, `ServiceOptions`, `StreamRequestOptions` | Defines the **`Service`** base class. Represents a cloud service endpoint (e.g. `storage.googleapis.com`). It initializes the authentication client (`GoogleAuth`), configures request interceptors, dynamically fetches/updates the GCP Project ID, appends correct telemetry headers (`User-Agent` & `x-goog-api-client`), and wraps requests through `util.makeAuthenticatedRequestFactory`. | +| **[`service-object.ts`](src/service-object.ts)** | `ServiceObject`, `ServiceObjectConfig`, `Interceptor`, `MetadataCallback` | Defines the **`ServiceObject`** class. Represents a single resource or entity under a service (like a Bucket, a BigQuery Dataset, or a Pub/Sub Topic). Extends `EventEmitter`. Provides generic CRUD method definitions (`create`, `get`, `exists`, `delete`, `getMetadata`, `setMetadata`). Delegates all HTTP/stream requests back to its parent `Service` or `ServiceObject` instance while prefixing URI paths with its own resource ID. Automatically promisified via `@google-cloud/promisify`. | +| **[`operation.ts`](src/operation.ts)** | `Operation` | Extends `ServiceObject` to handle **Long-Running Operations (LROs)**. Downstream cloud operations that are completed asynchronously by GCP servers (e.g., bulk imports, schema updates) inherit from this class. It listens for standard `complete` event handlers to automatically trigger an exponential background polling loop that fetches updated metadata until the operation's status is marked `done: true`. | +| **[`util.ts`](src/util.ts)** | `util`, `ApiError`, `PartialFailureError`, `ProgressStream`, `Abortable` | A rich collection of helper functions and custom error classes. Implements request authorization hooks, request decoration (which safely replaces raw `{{projectId}}` tokens with the active project ID across URIs, headers, and JSON payloads), response parsing, HTTP retry verification, Duplex stream piping, and User-Agent formatting. | + +--- + +## πŸ§ͺ Unit Test Suite File Map (`test/`) + +Unit tests are stored under `test/` and follow the exact name structure of the files under `src/`. They validate isolated library components offline using `Mocha` as the test runner, `Sinon` for method stubbing/sandboxing, and `proxyquire` for dependency injection overriding. + +| Test File | Target File | Detailed Test Scopes | +| :--- | :--- | :--- | +| **[`index.ts`](test/index.ts)** | `src/index.ts` | Verifies public export matching and type/class mappings. Ensures downstream clients have seamless access to the full core interface. | +| **[`service.ts`](test/service.ts)** | `src/service.ts` | Tests `Service` class instantiation, base configurations (`baseUrl`, `apiEndpoint`), specific Cloud Functions environment triggers (forces `forever: false`), telemetry header injection (`x-goog-api-client`), Request Interceptor ordering, dynamic project ID extraction/caching from the auth client, and correct path resolution in standard `request_` and `requestStream` handlers. | +| **[`service-object.ts`](test/service-object.ts)** | `src/service-object.ts` | Focuses on whitelisting of API methods on instantiation, full validation of `create`, `delete`, `exists`, `get`, `getMetadata`, and `setMetadata` lifecycles. Validates the `autoCreate` flow (which dynamically retries on Conflict `409` or retrieves on Missing `404` errors). Evaluates path formatting (stripping empty components, slash-trimming, supporting absolute URLs) and multi-layered interceptor chaining across parent-child class structures. | +| **[`operation.ts`](test/operation.ts)** | `src/operation.ts` | Assesses the polling loop for Long-Running Operations. Mocks internal operational check-points (`startPolling_`, `poll_`) using Sinon sandboxes. Verifies proper registration of the `complete` event listener, exponential polling timing, retry triggers, progress error propagation, and standard callback/promise returns. | +| **[`util.ts`](test/util.ts)** | `src/util.ts` | The largest and most comprehensive unit test file in the package. Validates utility routines:
β€’ **`ApiError` & `PartialFailureError`**: Error formatting, duplicate message filtering, multi-error array mapping, and parsing response JSON bodies.
β€’ **`handleResp`**: Structured error identification and extraction of status codes from HTTP response packets.
β€’ **`makeWritableStream`**: Multi-part form uploads, custom type boundary wrapping, progress notifications, and stream completion events.
β€’ **`makeAuthenticatedRequestFactory`**: Authentication client setups, active header mappings, query replacements, abort forwarding, and fallback projectId hooks.
β€’ **`shouldRetryRequest`**: Identification of retryable HTTP errors (408, 429, 500, 502, 503, 504) and DNS query address errors (`EAI_AGAIN`).
β€’ **`makeRequest`**: Custom retry algorithms, deprecated argument handling, readable/writable stream pipeline construction (GET vs POST request handling), and stream/request abort mechanics.
β€’ **`decorateRequest`**: Cleaning of utility pagination properties (`autoPaginate`, `objectMode`) and token substitution (`{{projectId}}`) inside JSON structures, arrays, queries, and URIs. | + +--- + +## πŸ§ͺ System & Integration Test Suite File Map (`system-test/`) + +System tests verify the functionality of the `@google-cloud/common` library using live Node.js web environments and compiler testing. + +| Integration File | Purpose & Test Scopes | +| :--- | :--- | +| **[`common.ts`](system-test/common.ts)** | Launches a local live Node.js HTTP mock server on `localhost:8118` to perform real-world integration checks:
β€’ **Request Pipeline**: Validates successful HTTP requests and response payloads handled by `Service.request`.
β€’ **Retry Mechanics**: Simulates server failures using code `408` and asserts that the service attempts requests exactly 4 times (1 initial + 3 retries) before returning the error.
β€’ **Timeout Backoffs**: Queries unresponsive mock endpoints and verifies that connection failure errors (like `ECONNREFUSED`) correctly respect exponential time backoff rules. | +| **[`install.ts`](system-test/install.ts)** | Validates package installation and TypeScript declaration compilation correctness:
β€’ Generates a release package tarball (`.tgz`) using `npm pack`.
β€’ Copies a dummy client fixture project (`system-test/fixtures/kitchen`).
β€’ Executes `npm install` within the fixture project using the local package tarball, validating that compile-time definitions (`.d.ts`) compile cleanly and are fully importable by client systems. | +| **[`fixtures/kitchen/`](system-test/fixtures/kitchen)** | A minimal testing project that acts as a consumer application. Contains `package.json`, `tsconfig.json`, and a `src/index.ts` importing public classes, methods, and types from `@google-cloud/common` to verify interface compilation validity. | + +--- + +## πŸ”§ Tooling & Configuration Files + +The root directory contains several configuration files that define the CI/CD pipelines, testing setup, and formatting guidelines for the repository: + +* **`.OwlBot.yaml` & `owlbot.py`**: Used by OwlBot (Google's automated template synthesis tool) to copy, post-process, and keep repository structure, dependencies, and config templates in sync with Google's central templates. +* **`.trampolinerc`**: Configures GCB (Google Cloud Build) Trampoline, which is used to orchestrate containerized CI builds, testing tasks, and release steps. +* **`.mocharc.js`**: Defines standard Mocha test runner options, including timeout values, file extensions, and reporters used during test execution. +* **`.nycrc`**: Configuration for `c8` test coverage metrics. Establishes thresholds and reporter formats for code execution analysis. +* **`.jsdoc.js` & `.compodocrc`**: Setup options for generation of developer-facing JSDoc HTML API reference documentation. +* **`tsconfig.json`**: Controls TypeScript compiler rules and output build directories (`build/src` and `build/test`). +* **`package.json`**: Specifies metadata, runtime engine criteria (e.g. `node >= 18`), NPM scripts (e.g. `compile`, `test`, `lint`, `system-test`), and external runtime dependencies (e.g. `google-auth-library`, `teeny-request`, `retry-request`). diff --git a/core/generator/gapic-generator-typescript/CODEMETA.md b/core/generator/gapic-generator-typescript/CODEMETA.md new file mode 100644 index 000000000000..0344f9cb7532 --- /dev/null +++ b/core/generator/gapic-generator-typescript/CODEMETA.md @@ -0,0 +1,412 @@ +# `@google-cloud/gapic-generator-typescript` β€” Codebase Metadata & Developer Guide +# *WARNING*: This file is AI generated and may contain inaccuracies. + +This document provides a comprehensive, developer-oriented overview of the `@google-cloud/gapic-generator-typescript` repository. It outlines the codebase's architectural goals, lists and describes its core components, maps out its directory structure, and explains the complete testing and baseline verification suite in detail. + +--- + +## 1. Architectural & Functional Overview + +The `@google-cloud/gapic-generator-typescript` package is a code generator used to compile **Protocol Buffer (`.proto`)** definitions into robust, production-ready **Google Cloud client libraries** for Node.js and modern JavaScript environments. + +It operates as a plugin for the standard protobuf compiler (`protoc`) or is run directly as a CLI tool using compiled protobuf file descriptors. The generated clients conform to the **Google API Improvement Proposals (AIPs)** and support multiple transport layers, including **gRPC**, **gRPC-Fallback (HTTP/1.1 JSON)**, and native **REST (Discovery-based)** interfaces. + +### Key Design Patterns +- **Metadata-Driven Code Generation**: The generator augments raw protobuf descriptors (`FileDescriptorProto`, `ServiceDescriptorProto`, `MethodDescriptorProto`) with custom Google APIs configurations (such as `google.api.http` annotations, long-running operations parameters, retry settings, client-side batching, and resource patterns) before passing them to a templates layer. +- **Nunjucks Template Orchestration**: Standard template engines (`Nunjucks`) define the layouts and source text for generated files (`src/v1/some_client.ts`, `package.json`, `tsconfig.json`, system-tests, READMEs). This ensures separation of generator parsing rules from the code design of the generated client. +- **Baseline-Based Verification**: Because client code must remain syntactically perfect and exactly match GCP guidelines, the generator is verified using **baselines**. The output of compiling test protobuf definitions is compared line-by-line against reference directories checked into the repository. + +### Code Generation Pipeline + +```mermaid +graph TD + A[Protobuf Source Files .proto] -->|protoc parser| B[FileDescriptorSet JSON/Protobuf] + B -->|CLI Input / Stdin| C[gapic-generator-typescript] + C -->|1. Parsing & Validation| D[Naming & ResourceDatabase] + D -->|2. Schema Augmentation| E[augmented Proto / Service / Method Models] + E -->|3. Templating Engine| F[Nunjucks Templater] + F -->|4. Output Formatting| G[Prettier & JSON Stringify] + G -->|5. Generated Files| H[Client Library Source Code] +``` + +--- + +## 2. Directory Map & Core Folders + +The repository is structured as follows: + +| Directory / Path | Purpose & Contents | +| :--- | :--- | +| [`baselines/`](baselines) | Checked-in reference output folders representing correct CommonJS client generations for showcase and official GCP APIs. | +| [`baselines-esm/`](baselines-esm) | Reference output folders representing correct modern ECMAScript Module (ESM) client generations. | +| [`protos/`](protos) | Contains static TypeScript type descriptors (`index.d.ts`, `protos.json`) generated from Google API definitions used at runtime. | +| [`rules_typescript_gapic/`](rules_typescript_gapic) | Custom Bazel build rules and macro files (`typescript_gapic.bzl`) integrating the generator with Bazel compilation targets. | +| [`templates/`](templates) | Nunjucks (`.njk`) templates defining source file layouts (TypeScript classes, package/linter configs, testing harnesses). | +| [`test-fixtures/`](test-fixtures) | Sample proto files, service configs, and JSON options representing edge cases and showcase API definitions. | +| [`typescript/src/`](typescript/src) | Core TypeScript implementation files for schema parsing, mapping, CLI input, and templates compilation. | +| [`typescript/test/`](typescript/test) | Unit tests, baseline verification drivers, and custom test runner utilities. | +| [`typescript/tools/`](typescript/tools) | Standalone utility scripts (e.g., `update-baselines.ts`) supporting the development workflow. | + +> [!NOTE] +> Directories prefixed with `.test-out-` (such as `.test-out-dlp/` or `.test-out-asset/`) are temporary directories generated dynamically during baseline verification and are excluded from version control. + +--- + +## 3. Core Configuration Files + +- **[`package.json`](package.json)**: Defines package properties, dependencies (e.g., `nunjucks`, `protobufjs`, `yargs`, `prettier`), and scripts: + - `compile`: Compiles TypeScript source in `typescript/` into runnable JavaScript inside the `build/` folder. + - `test`: Executes both the unit test suites and baseline integration tests. + - `baseline`: Runs baseline updates when template changes are verified. +- **[`tsconfig.json`](tsconfig.json)**: TypeScript compiler options directing the transpiler to output ECMAScript compatibility targets, resolve modern modules, and preserve typings. +- **[`BUILD.bazel`](BUILD.bazel) / [`WORKSPACE`](WORKSPACE) / [`MODULE.bazel`](MODULE.bazel)**: Configuration for the Bazel build pipeline, managing external workspace dependencies and compilation targets sandbox-style. + +--- + +## 4. Deep Dive: Source Code Components (`typescript/src/`) + +Core execution logic is located in [`typescript/src/`](typescript/src): + +```mermaid +graph LR + entry[gapic-generator-typescript.ts] --> gen[generator.ts] + gen --> parse[schema/api.ts] + parse --> proto[schema/proto.ts] + parse --> naming[schema/naming.ts] + parse --> db[schema/resource-database.ts] + parse --> retry[schema/retryable-code-map.ts] + gen --> temp[templater.ts] +``` + +### `gapic-generator-typescript.ts` +The **main command-line entry point**. It handles arguments parsing (e.g., output directory, templates config, package name, transport type, gRPC service config) using `yargs`. It loads compiled protobuf file descriptor paths and passes parsed options down to the code generator orchestrator. + +### `generator.ts` +The **core orchestration engine**. It coordinates: +1. Reading protobuf file descriptors from stdin or input arguments. +2. Instantiating the `API` parsing object to build a comprehensive schema graph. +3. Triggering the templating pipeline through the `Templater` class. +4. Formatting and post-processing all generated output files with `Prettier` and JSON serializers. + +### `templater.ts` +Manages **Nunjucks rendering**. It compiles template templates (configured in [`templates/`](templates)) using the augmented API metadata, translating them into actual client classes, helper files, configuration maps, and documentation. + +--- + +### The Schema Representation Layer + +To support templates, raw Protobuf schemas must be augmented with advanced structural details. This logic is organized inside `typescript/src/schema/`: + +#### [`schema/api.ts`](typescript/src/schema/api.ts) +Aggregates parsed resources and packages into a single representation of the API client. It initializes naming conventions, tracks and resolves resource maps, filters out common system helper packages, and tracks available API versions. + +#### [`schema/proto.ts`](typescript/src/schema/proto.ts) +The largest and most complex class. It maps gRPC structures into client features: +- **LRO (Long-Running Operations)**: Parses `google.longrunning.operationInfo` annotations to identify returned response and metadata types. +- **Pagination**: Implements Google AIP-158 pagination rules. Validates request and response fields (`page_token`, `page_size`, `next_page_token`, and repeated collection resources) to automatically decorate methods with paging wrappers. +- **Routing Parameters & HTTP Headers**: Maps standard parameters in `google.api.http` rules to default headers, and processes advanced `google.api.routing` annotations using matching regex to generate client-side routing keys. +- **Retry Configurations**: Maps status codes and timeouts into structured settings compatible with the `google-gax` library. +- **Service Mixins**: Processes options to auto-inject helpers like IAM Policy, Cloud Locations, and Long-Running Operations unless the client overrides them. + +#### [`schema/naming.ts`](typescript/src/schema/naming.ts) +Parses package and namespace hierarchy configurations (e.g., `google.cloud.speech.v1`) to extract valid JavaScript identifiers, client library class names, product versions, and publishing parameters. + +#### [`schema/resource-database.ts`](typescript/src/schema/resource-database.ts) +Indexes all resource patterns (`google.api.resource` and `google.api.resource_definition`). It: +- Maps resource types to specific path patterns. +- Handles complex multi-pattern entities by calculating unique disambiguated naming segments. +- Walks up resource segments to resolve hierarchical parent-child relationships. + +#### [`schema/retryable-code-map.ts`](typescript/src/schema/retryable-code-map.ts) +Manages mappings between gRPC status codes and snake_case parameter aliases (e.g., `idempotent` vs `non_idempotent`), compiling correct JSON retry blocks for client configurations. + +#### [`schema/comments.ts`](typescript/src/schema/comments.ts) +Extracts code-level documentation comments from protobuf elements, compiling cleanly formatted jsdoc summaries for generated methods. + +--- + +## 5. The Testing & Baseline Verification Suite + +The codebase features a two-tier testing structure: **Granular Unit Tests** (focusing on logical algorithms and configuration parsing) and **Baseline Integration Tests** (focusing on full-scale code generation). + +``` +typescript/test/ +β”œβ”€β”€ unit/ # Granular unit test suites +β”‚ β”œβ”€β”€ api.ts +β”‚ β”œβ”€β”€ naming.ts +β”‚ β”œβ”€β”€ proto.ts +β”‚ β”œβ”€β”€ resource-database.ts +β”‚ β”œβ”€β”€ retryable-code-map.ts +β”‚ └── util.ts +β”œβ”€β”€ baselines.ts # CommonJS baseline integration runner +β”œβ”€β”€ baselines-esm.ts # ESM baseline integration runner +β”œβ”€β”€ unit-test-runner.ts # Custom Mocha test orchestrator +└── util.ts # Baseline execution and diff utility +``` + +### 1. Unit Testing Infrastructure + +- **[`unit-test-runner.ts`](typescript/test/unit-test-runner.ts)**: Programmatically initializes a Mocha runner instance, scans the compiled `build/typescript/test/unit/` directory, dynamically loads the tests, and handles process codes. +- **[`unit/util.ts`](typescript/test/unit/util.ts)**: Validates global text transformations and duration calculations (`commonPrefix`, `seconds`, `milliseconds`, `capitalize`, `words`, casing maps, and URL path-template parsing regex). +- **[`unit/naming.ts`](typescript/test/unit/naming.ts)**: Verifies namespace parsing conventions. Asserts that the generator identifies versions, namespace groupings, and handles errors (such as package mismatch or invalid structure). +- **[`unit/resource-database.ts`](typescript/test/unit/resource-database.ts)**: Asserts correct indexing of resource descriptors. Tests multi-pattern registrations, pluralizations, custom slash delimiters (e.g. `taskId-taskName`), and parent-child tree generation. +- **[`unit/retryable-code-map.ts`](typescript/test/unit/retryable-code-map.ts)**: Verifies status codes mapping configurations. Tests hash caching for retry configurations and validates that custom retry definitions resolve to expected JSON layouts. +- **[`unit/api.ts`](typescript/test/unit/api.ts)**: Verifies the high-level API class initialization logic. Asserts proper behavior for standalone package generations (like IAM or Location services) and checks that missing required host options throw clear errors. +- **[`unit/proto.ts`](typescript/test/unit/proto.ts)**: Assertions for the complex parsing behavior inside `schema/proto.ts`. Verifies parameter extraction, regex generation from routing templates, pagination detection, LRO validations, and auto-populated request ID fields. + +--- + +### 2. Baseline Integration Testing + +Because code compilation is a deterministic process, integration testing relies on **baseline snapshots**. + +#### Integration Drivers +- **[`baselines.ts`](typescript/test/unit/baselines.ts)**: Driver defining baseline tests for all supported Google Cloud APIs using the standard CommonJS format. +- **[`baselines-esm.ts`](typescript/test/unit/baselines-esm.ts)**: Driver verifying that compiling libraries using the ECMAScript Modules (`format: 'esm'`) option results in ESM-compliant outputs. + +The baseline suite executes runs across representative GCP and Showcase APIs, verifying features including: +- **`bigquery-v2`**: Advanced queries and massive schemas. +- **`logging`**: High-throughput log delivery utilizing batching/bundling configurations. +- **`showcase`**: A testbed API verifying REST transport features, numeric enums, mixins control, and legacy proto loading. +- **`compute`**: Testing GCE REST-only discovery API interfaces (`diregapic: true`). +- **`deprecatedtest`**: Validates that APIs correctly mark methods and classes as `@deprecated`. +- **`routingtest`**: Verifies that complex, multi-parameter dynamic routing rule templates map to matching regex rules. + +#### Baseline Execution Engine: [`test/util.ts`](typescript/test/util.ts) +`test/util.ts` extends Mocha with a custom snapshot validator: +1. Receives a `BaselineOptions` block configuring compilation variables. +2. Wipes any previously generated output directory (under `.test-out-*`). +3. Formulates and spawns a child process to execute the compiler (`gapic-generator-typescript.js`) with target configurations. +4. Walks the newly generated directory recursively and compares each generated file line-by-line against the reference baseline inside `baselines/` (or `baselines-esm/`). +5. Highlights exact file paths and mismatched line coordinates on error, failing the suite if any discrepancy is found. + +--- + +## 6. Development Workflow & Utility Tools + +When modifying the templates in `templates/` or the parsing engine in `src/`, generated outputs will inevitably diverge from reference baseline expectations, causing `npm test` to fail. + +To propagate changes across all reference baseline directories, developers use the custom baseline updater tool via `npm run baseline`: + +### [`typescript/tools/update-baselines.ts`](typescript/tools/update-baselines.ts) + +This script programmatically automates baseline synchronization: +1. Scans the root directory and removes all obsolete trial outputs (`.test-out-*`). +2. Attempts to run `npm test`. If all baseline comparisons succeed, it exits early. +3. If comparisons fail, it identifies all generated trial folders, extracts the matching library name, and deletes the matching baseline reference folders under `baselines/` or `baselines-esm/`. +4. Copies the generated code to the baseline folder, adding `.baseline` suffixes to the files. +5. For `package.json` files, it copies them cleanly and sets up a relative symlink to `package.json.baseline`, ensuring compatibility with standard package managers and automated upgrade bots (like Renovate). + +**Command usage:** +```bash +# Build the generator and run baseline sync +npm run compile +npm run baseline +``` + +--- + +## 7. Nunjucks Templates Inventory (`templates/`) + +The generator relies entirely on **Nunjucks (`.njk`)** templates to separate compilation schemas from the visual design of the generated codebase. These templates define the structure, naming conventions, classes, configurators, and test harnesses for the compiled libraries. + +### Template Target Structures + +The `templates/` folder is organized into two parallel targets to support target compatibility targets: +1. **CommonJS (CJS) (`templates/cjs/`)**: Generates standard Node.js client modules using `require()` statements, CJS testing setups, and classic target builds. +2. **ECMAScript Modules (ESM) (`templates/esm/`)**: Generates modern ESM modules utilizing `import`/`export` syntax, transpiled bundle configurations, and ESM compilation pipelines. + +```mermaid +graph TD + T[templates/] --> CJS[cjs/typescript_gapic/] + T --> ESM[esm/typescript_gapic/] + + CJS --> CJS_Util["Shared Macros (_util.njk, _iam.njk, etc.)"] + CJS --> CJS_Conf["Config templates (package.json.njk, webpack.config.js.njk, etc.)"] + CJS --> CJS_Src["Source Entrypoints (src/index.ts.njk, src/$version/index.ts.njk)"] + CJS --> CJS_Client["Client Class (src/$version/$service_client.ts.njk)"] + CJS --> CJS_Test["Mocha Unit Tests (test/gapic_$service_$version.ts.njk)"] + + ESM --> ESM_Util["Shared Macros"] + ESM --> ESM_Conf["Config templates"] + ESM --> ESM_Src["Source Entrypoints"] + ESM --> ESM_Client["Client Class"] + ESM --> ESM_Test["Mocha Unit Tests"] +``` + +--- + +### Shared Macro Utilities & Mixins + +The following Nunjucks files define reusable utility macros loaded by main source templates to keep code generation modular: + +| File Path | Purpose & Description | Key Macros / Operations Defined | +| :--- | :--- | :--- | +| **`_license.njk`** | Injects consistent, standard Apache-2.0 copyright headers dynamically stamped with the copyright year into every source, config, and test file. | `license(copyrightYear)` | +| **`_namer.njk`** | Encapsulates naming helpers for variables, services, and package modules, initializing class descriptors cleanly. | `initialize(id, service)` | +| **`_util.njk`** | The central template library. Resolves protobuf schema models into type-safe TypeScript parameters, compiles routing configs, and generates docstring blocks. *(See mini-roadmap below)* | `printComments`, `buildHeaderRequestParam`, `initRequestWithHeaderParam`, `toInterface`, `typescriptType` | +| **`_iam.njk`** | Handles the standard IAM Policy API Mixin. Auto-generates JSDoc and client wrappers for checking and mutating service access policies. | `iamServiceMethods(service)` (`getIamPolicy`, `setIamPolicy`, `testIamPermissions`) | +| **`_locations.njk`** | Handles the Cloud Locations API Mixin. Auto-generates helper wrappers for querying service regions and data residency regions. | `locationServiceMethods(service)` (`getLocation`, `listLocations`) | +| **`_operations.njk`** | Handles the standard Long-Running Operations (LRO) Mixin. Generates stubs for managing operations execution lifecycles. | `operationsServiceMethods(service)` (`getOperation`, `cancelOperation`, `deleteOperation`) | + +--- + +### Configuration & Project Harness Templates + +These templates output the scaffolding needed to build, lint, test, and publish the target NPM package: + +- **`package.json.njk`**: Maps generator metadata into a fully structured `package.json` file containing all required production runtime libraries (e.g., `google-gax`, `protobufjs`), build utilities (`typescript`, `gts`, `mocha`), and lifecycle commands. +- **`tsconfig.json.njk` / `tsconfig.esm.json.njk`**: Emits project configuration targets for TypeScript, configuring standard path parameters, strict type checks, and module target compilations. +- **`webpack.config.js.njk` / `webpack.config.cjs.njk`**: Generates a ready-to-use Webpack bundle configuration enabling the client library to seamlessly execute in frontend browser environments utilizing gRPC fallback channels. +- **`README.md.njk`**: Auto-generates standard, copy-paste friendly Markdown documentation including: + - Quickstart guides detailing install commands. + - Clean authentication patterns (ADC setups). + - Quick reference tables detailing all service clients and methods. +- **`.jsdoc.js.njk` / `.jsdoc.cjs.njk`**: Emits configuration details pointing JSDoc pipelines to correctly compile code documentation pages. +- **`.mocharc.js.njk` / `.nycrc.njk`**: Bundles testing orchestrations directing Mocha and NYC code-coverage utilities to execute generated unit tests. + +--- + +### Module Entrypoints & Index Templates + +These templates establish clear module exports for modular imports: + +- **`src/index.ts.njk`**: The main entrypoint for the output library. Exports the full module, exposing all available API versions, namespace clients, and auxiliary proto models. +- **`src/$version/index.ts.njk`**: Aggregates and exports individual client classes and proto definitions specific to a single API release version (e.g. `v1` or `v2`). + +--- + +### Detailed Mini-Roadmap: `src/$version/$service_client.ts.njk` (1,181 lines) + +This is the primary code template responsible for emitting the **main TypeScript service client class**. It manages stub construction, channel orchestration, transports, fallback mechanisms, and client-side features like pagination and LRO handlers. + +#### Key Sections & Points of Interest + +```carousel +```typescript +// Phase 1: Imports & Setup (Lines 1 - 74) +// - Imports standard Node.js Stream helpers, path patterns, and google-gax modules. +// - Resolves proto loading formats (compiled static protos vs dynamic JSON descriptors). +// - Configures fallback HTTP/1.1 connections when native gRPC is unavailable. +``` + +```typescript +// Phase 2: Class Definition & Properties (Lines 75 - 108) +// - Extends class properties representing client options. +// - Declares stub instances, routing parameter paths, and mixin operations clients. +``` + +```typescript +// Phase 3: Constructor & Options Loader (Lines 110 - 408) +// - Formulates default endpoints and user-agent header metrics. +// - Instantiates custom gRPC and gRPC-Fallback channel structures. +// - Maps long-running operations (LRO), paging, and streaming descriptors. +``` + +```typescript +// Phase 4: initialize() & Stub Connection (Lines 410 - 507) +// - Orchestrates asynchronous server handshakes. +// - Decorates low-level gRPC stubs with client decorators (retries, timeouts, headers). +``` + +```typescript +// Phase 5: Unary Method Stubs (Lines 600 - 733) +// - Emits standard asynchronous functions matching protobuf RPC methods. +// - Automatically constructs header routing parameters and auto-injects UUID fields. +``` + +```typescript +// Phase 6: Streaming & LRO Implementations (Lines 734 - 933) +// - Emits duplex and simplex connection channels returning CancellableStream wrappers. +// - Generates LRO wrapper operations alongside GetOperation getters (checkProgress methods). +``` + +```typescript +// Phase 7: Auto-Pagination (Lines 934 - 1104) +// - Injects auto-pagination decorators for AIP-158 compliant list methods. +// - Exposes both stream channels (Stream) and standard async-iterators (Async). +``` + +```typescript +// Phase 8: Path Renderers & Channel Cleanup (Lines 1115 - 1181) +// - Emits regex matcher and renderer blocks parsing protobuf path segments. +// - Declares close() termination routines tearing down active connection channels. +``` +``` + +--- + +### Detailed Mini-Roadmap: `test/gapic_$service_$version.ts.njk` (1,613 lines) + +This template outputs the **comprehensive unit testing suite** verified under Mocha and Sinon. It creates stubs for every transport method, asserts routing parameters, verifies exception propagation, and exercises mixins without requiring actual network connections. + +#### Key Sections & Points of Interest + +```carousel +```typescript +// Phase 1: Setup, Imports & Proto Parsing (Lines 1 - 83) +// - Imports Sinon, assert, mocha, and the generated client module. +// - Resolves default field value lookup methods (getTypeDefaultValue) using the JSON proto definitions. +``` + +```typescript +// Phase 2: Sinon Stub Factories (Lines 84 - 201) +// - Defines generic mocks to inject into the testing stubs. +// - Exposes stub builders customized for each RPC flavor: +// - stubSimpleCall, stubServerStreamingCall, stubBidiStreamingCall, +// - stubClientStreamingCall, stubLongRunningCall, stubPageStreamingCall. +``` + +```typescript +// Phase 3: Client Core Feature Verification (Lines 203 - 365) +// - Asserts configuration parsing and default variable lookups. +// - Tests universeDomain ("googleapis.com") parsing overrides and endpoint builders. +// - Verifies basic initialize() and close() execution cycles. +``` + +```carousel +// Phase 4: Unary & Streaming Assertions (Lines 366 - 839) +// - Exercises unary method stubs, verifying that responses deep-equal stub results. +// - Tests error forwarding, request argument verification, and UUID auto-population. +// - Asserts Node Stream stream lifecycle methods on data/error events. +``` + +```typescript +// Phase 5: Pagination & Async Iterator Checks (Lines 840 - 1088) +// - Validates standard pagination call arguments. +// - Tests that stream wrappers receive correct configurations and route results cleanly. +// - Asserts that Async iterables return expected chunks iteratively under for-await loops. +``` + +```typescript +// Phase 6: GCP Mixin Mocking (Lines 1089 - 1567) +// - Stub-verifies mixin method integrations (IAMPolicy, Cloud Locations, and LROs). +// - Confirms that the mixin clients correctly forward payload requests and headers. +``` + +```typescript +// Phase 7: Path Template Rendering (Lines 1568 - 1612) +// - Generates assertions for path builder renderers (e.g. client.somePath()). +// - Exercises path segment regex parsing methods, ensuring parsed parameters match input segments. +``` +``` + +--- + +### Detailed Mini-Roadmap: `_util.njk` (502 lines) + +The primary shared utility macro suite. It drives JSDoc annotation compilation, request/response mock generation for unit tests, and routing parameter parsing. + +#### Key Sections & Points of Interest + +- **JSDoc Generation (Lines 21 - 261)**: + - `printComments`: High-level orchestrator for a method's JSDoc, printing descriptions, parameters, options, and type definitions. + - `printReturn`: Dynamically generates customized JSDoc `@returns` descriptions based on whether a method is paging, server-streaming, client-streaming, LRO, or standard unary. +- **Header Parameter Generation (Lines 262 - 298)**: + - `buildHeaderRequestParam`: Generates the code that builds the `x-goog-request-params` header dynamically. It handles both standard implicit routing parameters and advanced `google.api.routing` dynamic regex rules. +- **Unit Test Initializers (Lines 300 - 418)**: + - `initRequestWithHeaderParam`: Creates mock request messages for test suites, dynamically looking up default values to ensure exact regex matching inside routing tests. + - `initResponse` & `initPagingResponse`: Generates realistic stub payloads to feed Sinon stubs in unit tests. +- **Type Translators (Lines 419 - 502)**: + - `typescriptType` & `toInterface`: Translates raw protobuf descriptor types into their corresponding TypeScript equivalents (e.g. `TYPE_BYTES` -> `Buffer`, custom types -> `protos.some.Interface`). + diff --git a/core/packages/gax/CODEMETA.md b/core/packages/gax/CODEMETA.md new file mode 100644 index 000000000000..7d251e2f9d69 --- /dev/null +++ b/core/packages/gax/CODEMETA.md @@ -0,0 +1,140 @@ +# Google API Extensions (google-gax) Architecture Reference +# *WARNING*: This file is AI generated and may contain inaccuracies. + +`google-gax` (Google API Extensions) is the core middleware transport and orchestration layer for Google Cloud Client Libraries in Node.js. Its primary purpose is to bridge high-level, automatically generated GAPIC (Google API Programming Interface Client) libraries with low-level network protocols like **gRPC** (`@grpc/grpc-js`) and **HTTP/1.1 JSON-REST Fallback** (`gaxios`). + +By providing a common set of behaviorsβ€”including exponential retries, request bundling (batching), automatic pagination unrolling, streaming duplex proxying, and long-running operation polling, `google-gax` ensures that Google Cloud clients behave consistently, efficiently, and resiliently under all network conditions. + +--- + +## 1. High-Level Request Lifecycle + +The core pattern in GAX is a **Descriptor-Driven Call Orchestration**. Client libraries do not make raw gRPC calls; instead, they invoke wrapped functions generated by `createApiCall`. Based on settings and custom descriptors supplied at call-creation time, `google-gax` dynamically intercepts and augments the request. + +```mermaid +graph TD + Client[GAPIC Client Call] --> CAC[createApiCall] + CAC --> Settings[Merge CallOptions & CallSettings] + Settings --> APICallerFactory[apiCaller.ts: createAPICaller] + + APICallerFactory -->|Normal Descriptor| NormalCaller[normalApiCaller.ts] + APICallerFactory -->|Bundle Descriptor| BundleCaller[bundleApiCaller.ts] + APICallerFactory -->|Page Descriptor| PagedCaller[pagedApiCaller.ts] + APICallerFactory -->|Stream Descriptor| StreamCaller[streamApiCaller.ts] + APICallerFactory -->|Long-Running Descriptor| LROCaller[longRunningApiCaller.ts] + + NormalCaller --> UnaryRPC[Unary gRPC / REST Call] + BundleCaller --> BundleExec[bundleExecutor.ts: Bundle Request] + PagedCaller --> PagingExec[pageDescriptor.ts: Fetch Pages] + StreamCaller --> StreamProxy[streaming.ts: Duplexify Wrapper] + LROCaller --> LROExec[longrunning.ts: Poll Operation] + + UnaryRPC --> Retries[retries.ts: Exponential Backoff] + Retries --> Channel[gRPC / Fallback HTTP Channel] +``` + +--- + +## 2. Core Module Map + +The library's core codebase resides within the `src/` directory. Below is an itemized map of each module's role and design pattern. + +| Module / Directory | File | Primary Responsibility | Key Design Patterns & Dependencies | +| :--- | :--- | :--- | :--- | +| **Call Orchestration** | [createApiCall.ts](src/createApiCall.ts) | Central factory that wraps low-level RPCs with timeout, retry, and descriptor-driven behaviors. | Factory Pattern, Decorator Pattern | +| | [apiCaller.ts](src/apiCaller.ts) | Defines the `APICaller` interface and dispatches calls to specialized callers based on the descriptor type. | Factory Method Pattern | +| | [descriptor.ts](src/descriptor.ts) | Base interface for `Descriptor` and implementations for specialized call markers. | Strategy Pattern | +| | [call.ts](src/call.ts) | Manages asynchronous call state, response/error propagation, and explicit request cancellation. | Promise Wrap, Observer Pattern | +| **Unary Calls** | [normalCalls/normalApiCaller.ts](src/normalCalls/normalApiCaller.ts) | Handles default single-request single-response (unary) calls. | Direct Execution Wrapper | +| | [normalCalls/retries.ts](src/normalCalls/retries.ts) | Implements exponential backoff retry loops using canonical error codes. | Retry Loop, Backoff Strategy | +| | [normalCalls/timeout.ts](src/normalCalls/timeout.ts) | Computes and attaches hard deadlines to outgoing RPC metadata. | Deadline Scheduling | +| **Request Bundling** | [bundlingCalls/bundleDescriptor.ts](src/bundlingCalls/bundleDescriptor.ts) | Extracts request elements and packs them using discriminator fields. | Strategy Pattern | +| | [bundlingCalls/bundleApiCaller.ts](src/bundlingCalls/bundleApiCaller.ts) | Delegates execution of single requests to the shared batch executor. | Client-Facing Proxy | +| | [bundlingCalls/bundleExecutor.ts](src/bundlingCalls/bundleExecutor.ts) | Batches multiple calls into a single request, enforcing thresholds (count, size, delay). | Task Queue, Scheduler, Timer-based Flusher | +| **Pagination** | [paginationCalls/pageDescriptor.ts](src/paginationCalls/pageDescriptor.ts) | Manages token-based pagination. Spawns standard and async-iterable streams. | Iterator & Generator Patterns | +| | [paginationCalls/pagedApiCaller.ts](src/paginationCalls/pagedApiCaller.ts) | Unrolls pages recursively using a collector to aggregate all items. | Recursion, Aggregation | +| **Streaming** | [streamingCalls/streamApiCaller.ts](src/streamingCalls/streamApiCaller.ts) | Prepares streaming calls and initializes the proxy stream. | Stream Interceptor | +| | [streamingCalls/streaming.ts](src/streamingCalls/streaming.ts) | Wraps gRPC streams into robust Duplex proxies, supporting server-streaming retries with resumption keys. | `duplexify`, Retry Proxy, Event Forwarding | +| **Long-Running Ops** | [longRunningCalls/longRunningDescriptor.ts](src/longRunningCalls/longRunningDescriptor.ts) | Describes unpackers (decoders) and Operations Clients for LROs. | Descriptor strategy | +| | [longRunningCalls/longRunningApiCaller.ts](src/longRunningCalls/longRunningApiCaller.ts) | Wraps LRO initiation calls, yielding custom `Operation` polling objects. | Client-Facing Proxy | +| | [longRunningCalls/longrunning.ts](src/longRunningCalls/longrunning.ts) | Manages standard event/promise-based LRO state polling with exponential backoff. | Polling Loop, EventEmitter | +| **Transport & Fallback** | [grpc.ts](src/grpc.ts) | Standard Node.js transport setup. Manages channels, client certificates (mTLS), and proto loading. | Channel Management, mTLS Provider | +| | [fallback.ts](src/fallback.ts) | Core browser-compatible/REST fallback engine interface. | Transport Adapter | +| | [fallbackRest.ts](src/fallbackRest.ts) | Encodes request objects to REST JSON payloads and decodes response buffers back to objects. | `proto3-json-serializer` | +| | [fallbackServiceStub.ts](src/fallbackServiceStub.ts) | Generates mock gRPC stubs that run HTTP requests via `gaxios`, handling streaming and legacy 204 responses. | Fetch Stub, Adapter | +| | [transcoding.ts](src/transcoding.ts) | Implements Google Cloud Transcoding specifications (HTTP path-matching, body placement, and query mapping). | Transcoder | +| | [streamArrayParser.ts](src/streamArrayParser.ts) | Parses continuous REST JSON stream arrays chunk-by-chunk. | Transform Stream | + +--- + +## 3. Deep-Dive: Architectural Patterns + +### A. Call Orchestration (`createApiCall.ts` & `call.ts`) +GAPIC clients generate calls using `createApiCall(func, settings, descriptor)`. +- **Settings Merging**: `CallSettings` are constructed by merging user-provided `CallOptions` with the default configuration. If the user supplies a custom timeout, it automatically overrides retry backoff constraints. +- **Cancellable Execution**: The request execution state is tracked using `OngoingCall` or `OngoingCallPromise`. When `cancel()` is triggered, GAX immediately propagates the cancel request down to the active gRPC call/fetch stub and rejects the pending promise with a `CANCELLED` code error. + +### B. Resilient Unary Calls & Backoff (`normalCalls/`) +When an API call is executed by `NormalApiCaller`, it wraps the call in a retry loop: +- **Deadline Calculation**: If a timeout is configured, `timeout.ts` computes the exact absolute millisecond deadline and injects it into the gRPC metadata. +- **Backoff Loop**: If the request fails with an error code listed in `RetryOptions.retryCodes` (e.g., `UNAVAILABLE`), GAX calculates a randomized delay using an exponential multiplier up to `maxRetryDelayMillis`. +- **Constraint**: If the total execution duration exceeds `totalTimeoutMillis` or the retry count exceeds `maxRetries`, the retry loop yields a `DEADLINE_EXCEEDED` error. + +### C. Request Bundling (`bundlingCalls/`) +Bundling groups multiple individual unary requests into a single batched request to reduce network overhead. +- **Batching Execution**: Client calls are intercepted by `BundleApiCaller`, which extracts individual elements using the `BundleDescriptor` and queues a new `Task` inside `BundleExecutor`. +- **Flush Strategy**: Tasks are processed and sent to the server when: + 1. The element count hits `element_count_threshold`. + 2. The accumulated byte size hits `request_byte_threshold` (computed using `createByteLengthFunction`). + 3. The delay timer hits `delay_threshold_millis`. +- **Task Partitioning**: Once the batched RPC returns, the `BundleExecutor` maps the results back to the original callers' callbacks. + +### D. Automatic Pagination (`paginationCalls/`) +For API methods that return paginated lists, `PagedApiCaller` automates results collection. +- **Stream / Iterator Delivery**: The caller exposes an automatic page unrolling mechanism. If `autoPaginate` is enabled, GAX recursively retrieves pages until the end of the collection, flattening the repeated fields into a single array or stream. +- **Manual Iteration**: If `autoPaginate` is set to `false`, the library directly exposes page tokens, allowing developers to control pagination manually. + +### E. Dynamic Stream Proxying & Resumption (`streamingCalls/`) +Streaming calls utilize `StreamProxy` (extending `duplexify`) to wrap client, server, or bidirectional streams. +- **Backpressure & Forwarding**: It proxies raw gRPC events (`data`, `metadata`, `status`, `response`) to the consumer, preserving stream backpressure. +- **Stream Resumption**: If server-streaming retries are enabled, `newStreamingRetryRequest` monitors stream errors. Upon failure, it uses `getResumptionRequestFn` to generate a new starting request argument (e.g., adjusting the resumption token offset) and seamlessly creates a new underlying stream. + +> [!IMPORTANT] +> GAX forces `autoPaginate` to `false` in streaming methods. Streaming retries are supported for server-streaming calls only. Generic GAX-level streaming retries are highly effective for **unidirectional read streams** (like Bigtable read rows or Cloud Storage downloads) where a cursor can be cleanly offset. However, for **bidirectional state-synchronization channels** (like Pub/Sub StreamingPull or Firestore Listen), the application-level library must manage the stream lifecycle manually. This ensures that flow-control states, lease extensions, and ack queues remain perfectly synchronized with the server across connection restarts. + +### F. Long-Running Operations (`longRunningCalls/`) +LRO calls do not immediately return results; they return an `Operation` tracking status: +- **Complete Hook Listener**: Polling is entirely event-driven. Registering a `complete` listener on the `Operation` immediately kicks off `startPolling_()`. When there are no more listeners, polling automatically ceases. +- **Polling Loop**: The operation queries `getOperation` using exponential backoff. It emits `'progress'` when the metadata updates and fires `'complete'` with decoded results once `done === true`. + +### G. HTTP/JSON-REST Fallback (`fallback.ts` & `transcoding.ts`) +For environments lacking native gRPC support (like browsers), GAX implements a JSON over HTTP/1.1 transport layer. +- **HTTP Transcoding**: Guided by `google.api.http` proto annotations, `transcoding.ts` matches and injects request fields directly into URLs (e.g., mapping `{name=projects/*}` into path variables), encodes body fields (using the `body` directive), and flattens remaining parameters into query string variables. +- **Stream Parser**: For server-streaming calls, `StreamArrayParser` reads incoming REST chunks, tracks JSON brace levels to isolate individual objects, and decodes them using `decodeResponse`. + +--- + +## 4. Testing Strategy + +The `google-gax` test suite is split into granular unit tests and cross-environment system tests. + +### A. Unit Tests (`test/unit/`) +These verify specific modular logic without requiring actual cloud connections. + +| Test File | Coverage Area | +| :--- | :--- | +| [apiCallable.ts](test/unit/apiCallable.ts) | Validates the base `createApiCall` factory, timeout injections, settings overrides, and cancel propagation. | +| [bundling.ts](test/unit/bundling.ts) | Tests bundling thresholds (byte limit, element limit, delay timer) and error propagation to batched tasks. | +| [googleError.ts](test/unit/googleError.ts) | Exercises gRPC status binary parser and HTTP/REST error mapping (extracting `ErrorInfo` metadata). | +| [grpc-fallback.ts](test/unit/grpc-fallback.ts) | Validates fallback REST stubs, custom headers, mTLS domain restrictions, and browser fallback. | +| [longrunning.ts](test/unit/longrunning.ts) | Exercises event-driven LRO polling loops, cancel actions, and promise resolution. | +| [pagedIteration.ts](test/unit/pagedIteration.ts) | Tests recursive page unrolling, async iterators, and manual page-token propagation. | +| [transcoding.ts](test/unit/transcoding.ts) | Validates path mapping, JSON body placement (`body: "*"`), and query param compilation. | +| [streamArrayParser.ts](test/unit/streamArrayParser.ts) | Tests chunked JSON array stream parsing, string escape handling, and token isolation. | +| [streaming.ts](test/unit/streaming.ts) | Verifies duplex proxying, event propagation, status forwarding, and server streaming retries. | + +### B. System Tests (`test/system-test/`) +System tests prove GAX's compatibility against real client libraries and endpoints. + +- **Integration Verification ([test.clientlibs.ts](test/system-test/test.clientlibs.ts))**: + Rather than relying on static mock tests, the system test dynamically packs the local code into a tarball (`npm pack`). It then clones a real, active client library repository (like the `speech` client, which covers unary, streaming, and LRO calls), rewires its dependencies to use the local `google-gax` tarball, runs a full suite of system tests, and verifies that no backward-compatibility regressions occurred. diff --git a/core/packages/gaxios/CODEMETA.md b/core/packages/gaxios/CODEMETA.md new file mode 100644 index 000000000000..a313e6114ee9 --- /dev/null +++ b/core/packages/gaxios/CODEMETA.md @@ -0,0 +1,131 @@ +# Gaxios Codebase Metadata & Architecture +# *WARNING*: This file is AI generated and may contain inaccuracies. + +Gaxios is a robust HTTP request client for Node.js and browser environments. It provides an `axios`-like developer interface implemented on top of native `fetch` (with a fallback to `node-fetch` in older environments). + +Gaxios is tailored specifically for the Google Cloud Node.js ecosystem (as a core dependency of libraries like `google-auth-library`, `gcp-metadata`, and others). It delivers specialized capabilities for resilient networking, including: +- **Advanced Retries**: Automatic request retries with configurable backoff rules. +- **Exponential Backoff**: Multiplier-driven delay calculation to avoid flooding servers. +- **Interceptors**: Asynchronous hooks for request and response processing. +- **Log Redaction**: Built-in protection against accidental credential leakage in logs. +- **Advanced Agents**: First-class support for HTTP/HTTPS proxies (including wildcard bypass rules) and Mutual TLS (mTLS). + +--- + +## πŸ—οΈ Request & Retry Pipeline Architecture + +The diagram below illustrates the lifecycle of an HTTP request made via Gaxios: + +```mermaid +flowchart TD + A[Client Request] --> B["#prepareRequest()"] + B --> C[Merge Defaults & Options] + C --> D[Configure Proxies / mTLS / Timeouts / Signals] + D --> E[Apply Request Interceptors] + E --> F[Dispatch Request via Adapter] + F --> G{Response Status Valid?} + G -- Yes --> H[Apply Response Interceptors] + H --> I[Resolve with GaxiosResponse] + G -- No --> J[AIP-193 Error Extraction] + J --> K[Throw GaxiosError] + K --> L{Should Retry?} + L -- Yes --> M[Calculate Exponential Backoff Delay] + M --> N[Increment currentRetryAttempt] + N --> O[Trigger onRetryAttempt Callback] + O --> P[Sleep / Wait for Backoff] + P --> B + L -- No --> Q[Apply errorRedactor] + Q --> R[Reject GaxiosError] +``` + +--- + +## πŸ“‚ Directory Structure + +``` +gaxios/ +β”œβ”€β”€ src/ # Core implementation source files +β”œβ”€β”€ test/ # Unit tests & testing assets (fixtures) +β”œβ”€β”€ system-test/ # Integration & package installation tests +β”œβ”€β”€ browser-test/ # Browser-level verification tests via Karma +β”œβ”€β”€ utils/ # Build and ESM configuration utilities +└── [configs] # Build, linting, and formatting configurations +``` + +--- + +## πŸ“ File Registry + +### πŸš€ Core Source Code (`src/`) + +| File | Type | Description | +| :--- | :--- | :--- | +| **[index.ts](src/index.ts)** | Entrypoint | Exposes the primary public API surfaces. Exports classes like `Gaxios`, `GaxiosError`, `GaxiosInterceptorManager` and associated interfaces. Creates and exports the default shared `instance` along with the convenient `request` shorthand. | +| **[common.ts](src/common.ts)** | Types & Errors | Defines types and interfaces (e.g. `GaxiosOptions`, `GaxiosResponse`, `RetryConfig`). Implements `GaxiosError` with support for standard nested `.cause` and `instanceof` checks across library versions. Provides `defaultErrorRedactor` to scrub credentials, tokens, and client secrets from logs. | +| **[gaxios.ts](src/gaxios.ts)** | Core Logic | Implements the central `Gaxios` class. Manages options normalization (`#prepareRequest`), interceptor execution, proxy mapping/exclusions (`#urlMayUseProxy`), mutual TLS (mTLS) setup, and multipart related payloads chunking (`getMultipartRequest`). | +| **[interceptor.ts](src/interceptor.ts)** | Hook System | Defines the `GaxiosInterceptor` hook model and implements `GaxiosInterceptorManager` (inheriting from standard `Set`) to support registering and clearing async request/response handlers. | +| **[retry.ts](src/retry.ts)** | Retry Strategy | Handles retry decision-making (`getRetryConfig`, `shouldRetryRequest`) and calculates delay timings (`getNextRetryDelay`) via multiplier-based exponential backoff, capping at defined limits or timeouts. | +| **[util.cts](src/util.cts)** | Package Utility | A CommonJS script that reads the package `name` and `version` directly from `package.json` to synchronize error markings and metadata. | + +--- + +### πŸ§ͺ Unit Tests (`test/`) + +Unit tests use [mocha](https://mochajs.org/) as the test runner, [nock](https://github.com/nock/nock) to mock HTTP traffic, and [sinon](https://sinonjs.org/) for mocks and stubs. + +| File | Target / Coverage | +| :--- | :--- | +| **[test.index.ts](test/test.index.ts)** | Ensures key exports (`Gaxios`, `GaxiosError`, `GaxiosInterceptorManager`) are successfully resolved from the library package entrypoint. | +| **[test.retry.ts](test/test.retry.ts)** | Exercises the retry and backoff state machine. Tests: | +| **[test.getch.ts](test/test.getch.ts)** | The central test suite for all core request operations. Tests: | + +#### Test Fixtures (`test/fixtures/`) +- **[fake.cert](test/fixtures/fake.cert)**: A mock SSL certificate used to test Mutual TLS (mTLS) HTTPS agent creation. +- **[fake.key](test/fixtures/fake.key)**: A mock private key used to test Mutual TLS (mTLS) HTTPS agent creation. + +--- + +### πŸ“¦ Integration & Installation Tests (`system-test/`) + +System tests verify packaging compatibility, dependency structure, and bundler support. + +| File / Directory | Description | +| :--- | :--- | +| **[test.install.ts](system-test/test.install.ts)** | Executes a multi-step integration workflow: | +| **[system-test/fixtures/sample/](system-test/fixtures/sample/)** | Mock client project containing configuration (`package.json`, `tsconfig.json`), a webpack configuration (`webpack.config.js`), and a source file (`src/index.ts`) importing and initiating a Gaxios request to test build compatibility. | + +--- + +### πŸ’» Browser Tests (`browser-test/`) + +Browser tests confirm execution correctness directly in front-end client contexts. + +| File | Description | +| :--- | :--- | +| **[browser-test-runner.ts](browser-test/browser-test-runner.ts)** | Starts a local, CORS-compliant Express server on port `7172` that parses standard GET, query-string, and multipart POST payloads. Automatically runs **Karma** (`karma start`) to run unit tests in a real browser and stops the Express server upon completion. | +| **[test.browser.ts](browser-test/test.browser.ts)** | Contains unit tests executing in the browser via Karma. Asserts that standard requests, query parameters, custom fetch overrides (`window.fetch`), and complex multipart related uploads work natively in browser runtimes. | + +--- + +### πŸ› οΈ Build Utilities (`utils/`) + +- **[enable-esm.mjs](utils/enable-esm.mjs)**: A post-build utility script that writes a `{"type": "module"}` manifest inside the `./build/esm/` distribution folder to ensure Node.js interprets target build outputs as ES modules. + +--- + +### βš™οΈ Configuration Registry + +The root of the package contains multiple configuration files defining the compilation, verification, and style systems: + +- **`package.json`**: Lists dependencies, build directives, runtime engines requirements, and CJS/ESM exports paths. +- **`tsconfig.json`**, **`tsconfig.base.json`**, **`tsconfig.cjs.json`**: TypeScript configuration files setting up target definitions and module output generation. +- **`webpack.config.js`**, **`webpack-tests.config.js`**: Webpack compilation rules for bundling Gaxios and its test files. +- **`karma.conf.js`**: Setups browser test engines, plugins, and execution targets. +- **`.compodocrc`**: Directs the Compodoc engine for generating static documentation. +- **`.eslintignore`**, **`.eslintrc.js`**, **`.eslintrc.json`**: ESLint rules and scopes. +- **`.jsdoc.js`**: Configures JSDoc generation. +- **`.mocharc.js`**: Standard runner options for mocha unit tests. +- **`.nycrc`**: NYC coverage thresholds and file inclusions. +- **`.prettierrc.js`**, **`.prettierignore`**: Prettier rules enforcing stylistic alignment with the Google JS Style Guide. +- **`.repo-metadata.json`**: Specifies API classifications, stability level (`stable`), and library categorizations. +- **`linkinator.config.json`**: Configurations for checking and avoiding broken markdown hyperlinks. diff --git a/core/packages/gcp-metadata/CODEMETA.md b/core/packages/gcp-metadata/CODEMETA.md new file mode 100644 index 000000000000..8daa94d556e2 --- /dev/null +++ b/core/packages/gcp-metadata/CODEMETA.md @@ -0,0 +1,147 @@ +# Codebase Metadata: GCP Metadata Client (`gcp-metadata`) +# *WARNING*: This file is AI generated and may contain inaccuracies. + +This directory houses the `@google-cloud/gcp-metadata` (or `gcp-metadata`) package, a lightweight, high-performance Node.js client designed to query the Google Cloud Platform (GCP) Metadata Server. + +The GCP Metadata Server is a vital resource when running Node.js workloads inside GCP environments (such as Google Compute Engine, Google Kubernetes Engine, Google Cloud Run, Google Cloud Functions, and Google App Engine). It serves runtime configuration, instance identifiers, service account OAuth tokens, project details, and more. + +--- + +## Architectural Overview & Key Features + +1. **Automatic GCP Environment Detection**: + Automatically probes local host systems, network configurations, and environment variables to deduce if the code is executing on a GCP compute platform. +2. **Optimized Connection Racing**: + To maximize speed and resilience across varying DNS setups, the library races an IP-based query (`http://169.254.169.254`) against a DNS-based query (`http://metadata.google.internal.`) using `Promise.any`. This prevents unnecessary delays in environments with slow DNS resolution. +3. **Developer-friendly Fail-Fast Locally**: + If the library detects it is not running on Google Cloud (e.g., on a developer's workstation), it uses a brief 3-second timeout for network checks instead of waiting indefinitely or relying on long OS-level connection timeouts. +4. **High-Precision Large Number Parsing**: + Employs `json-bigint` to safely parse metadata server JSON responses that contain extremely large numbers (like long GCE project/instance IDs) without loss of precision. +5. **Caching**: + Memoizes environment detection and metadata server availability checks to reduce redundant HTTP requests and minimize overhead. + +--- + +## Directory Structure & File Map + +```mermaid +graph TD + src_index[src/index.ts] --> src_residency[src/gcp-residency.ts] + test_index[test/index.test.ts] --> src_index + test_residency[test/gcp-residency.test.ts] --> src_residency + test_residency --> test_utils[test/utils/gcp-residency.ts] + test_index --> test_utils + sys_test[system-test/system.ts] --> hook_fixture[system-test/fixtures/hook] + sys_test --> build_fixture[system-test/fixtures/cloudbuild] + kitchen_test[system-test/kitchen.test.ts] --> kitchen_fixture[system-test/fixtures/kitchen] + sample_test[samples/test/test.js] --> sample_quick[samples/quickstart.js] +``` + +### πŸ“ Core Source Files (`src/`) + +* #### **`src/index.ts`** + * **Purpose**: The primary API module and client entry point. + * **Key APIs**: + * `instance(options)`: Fetches metadata for the current Google Compute Engine (GCE) instance. + * `project(options)`: Fetches metadata for the current Google Cloud project. + * `universe(options)`: Fetches universe-specific settings (e.g. the universe domain). + * `bulk(properties)`: Concurrently resolves multiple metadata values via `Promise.all`. + * `isAvailable()`: Checks if the metadata server is accessible. Memoizes results and uses connection racing (`Promise.any`) between the IP address (`169.254.169.254`) and DNS address (`metadata.google.internal.`). + * `resetIsAvailableCache()`: Utility to clear the memoized availability check cache. + * `getGCPResidency()` / `setGCPResidency(value)`: Accesses or overrides cached GCP residency flags. + * `requestTimeout()`: Calculates the appropriate query timeout (0ms on GCP, 3000ms locally). + * **Technical Details**: Implements custom HTTP options validation, and leverages `json-bigint` for loss-free numeric JSON parsing. + +* #### **`src/gcp-residency.ts`** + * **Purpose**: Low-level system and environment analyzer to determine if the runtime platform is GCP. + * **Key APIs**: + * `isGoogleCloudServerless()`: Inspects process environment variables (`CLOUD_RUN_JOB`, `FUNCTION_NAME`, `K_SERVICE`) to detect Google Cloud Run or older/newer Google Cloud Functions. + * `isGoogleComputeEngineLinux()`: Probes Linux DMI tables (specifically `/sys/class/dmi/id/bios_vendor` and `/sys/class/dmi/id/bios_date`) to check if the BIOS vendor matches `Google`. + * `isGoogleComputeEngineMACAddress()`: Inspects active local network interface MAC addresses. Returns `true` if any interface's MAC address matches the `/^42:01/` GCE prefix pattern. + * `isGoogleComputeEngine()`: Combines the GCE Linux BIOS and GCE MAC address checks. + * `detectGCPResidency()`: Master check aggregating serverless and Compute Engine verification. + +--- + +### πŸ“ Test Files (`test/`) + +* #### **`test/index.test.ts`** + * **Purpose**: Comprehensive unit test suite validating all options, query routines, fallback triggers, and environment caching logic in the core index API. + * **Coverage**: + * *Request/Response validation*: Assures custom query parameters (`params`), custom headers, and paths are generated properly. + * *Header Security checks*: Validates that missing or incorrect `Metadata-Flavor: Google` response headers cause expected RangeErrors. + * *JSON Parsing*: Ensures high-precision bigints and nested numbers within responses are parsed perfectly using `json-bigint`. + * *Local Overrides*: Confirms the client respects custom overrides via `GCE_METADATA_IP`, `GCE_METADATA_HOST`, and `METADATA_SERVER_DETECTION` environment variables. + * *Racing & Timeouts*: Simulates various delays and failures between primary and secondary hostnames, proving that `isAvailable()` correctly returns the first successful response and fails fast under heavy network lags. + * *Caching*: Confirms `isAvailable()` caches its promise so rapid parallel invocations perform only a single outbound network request. + +* #### **`test/gcp-residency.test.ts`** + * **Purpose**: Unit tests focusing entirely on validating the environment analyzer logic inside `gcp-residency.ts`. + * **Coverage**: + * Tests Cloud Run and Cloud Functions environment detection by mocking process environment variables. + * Asserts Linux GCE BIOS file detections by mocking filesystem access and DMI vendor files. + * Asserts MAC address verification by mocking the `os.networkInterfaces()` responses to mimic both standard and GCE-prefixed MAC interfaces. + +* #### **`test/utils/gcp-residency.ts`** + * **Purpose**: Sandbox helper utility class (`GCPResidencyUtil`) for unit testing. + * **Key APIs**: + * `setGCENetworkInterface(isGCE)`: Stubs OS network interfaces to provide a mock MAC address. + * `setGCEPlatform(platform)`: Stubs OS platform return values. + * `setGCELinuxBios(isGCE)`: Stubs filesystem stats and file reads for the Linux DMI/BIOS vendor files. + * `removeServerlessEnvironmentVariables()`: Cleans out GCF/Cloud Run variables. + * `setNonGCP()`: Utility to stub a complete local machine non-GCP sandbox environment. + +--- + +### πŸ“ System Tests (`system-test/`) + +* #### **`system-test/system.ts`** + * **Purpose**: End-to-end integration tests validating actual connectivity and authenticity against real GCP metadata servers inside active Google Cloud runtimes. + * **Coverage**: + * *Cloud Functions (Gen 2)*: Dynamically bundles the local package using `npm pack`, copies the package to a function fixture, deploys it using `gcloud functions deploy`, and polls real-time Cloud Logging using `gcloud logging read` to verify the metadata server was detected (`isAvailable=true`). + * *Cloud Build*: Uses `gcbuild` to execute a Cloud Build step with a fixture, verifying that the metadata server resolves successfully inside the build environment. + * *Pruning / Garbage Collection*: Includes logic to automatically scan and delete old test functions left over from prior test runs, keeping resource usage clean. + +* #### **`system-test/kitchen.test.ts`** + * **Purpose**: Packaging and TypeScript compilation sanity test (a "kitchen sink" test). + * **Coverage**: + * Compresses the codebase into a tarball and copies it to a clean temporary environment containing the `kitchen` TypeScript fixture. + * Runs `npm install` and compiles/runs the fixture to verify that downstream applications can successfully consume and reference the compiled type declarations (`.d.ts`) without syntax errors. + +--- + +### πŸ“ System Test Fixtures (`system-test/fixtures/`) + +* #### **`system-test/fixtures/cloudbuild/`** + * `index.js`: Executed inside the mock Cloud Build process. Queries `isAvailable()`, fetches default service account tokens, and reads service account details recursively. + * `cloudbuild.yaml`: Defines the steps GCB uses to run the node fixture file. + +* #### **`system-test/fixtures/hook/`** + * `index.js`: The entry point script for the GCF deployment test. Exposes the HTTP endpoint `getMetadata` which outputs results of `.isAvailable()`, `.instance()`, and recursive service accounts list. + +* #### **`system-test/fixtures/kitchen/`** + * Contains `tsconfig.json` and package configuration to verify downstream type compatibility. + +--- + +### πŸ“ Samples & Examples (`samples/`) + +* #### **`samples/quickstart.js`** + * **Purpose**: Clean, simple example code showcasing how to integrate the library, perform availability checks, and safely fetch instance and project metadata. + +* #### **`samples/test/test.js`** + * **Purpose**: Unit/integration test verifying the quickstart sample works as intended. + * **Coverage**: + * Spins up a local Node.js HTTP server mimicking the metadata server. + * Launches `quickstart.js` as a child process, directing it to the mock server via the `GCE_METADATA_HOST` environment variable, and asserts that the stdout output prints "Is available: true". + +--- + +## Configuration & Manifest Files + +* `package.json`: Configures package targets, CommonJS module format, and development scripts (`compile`, `lint`, `test`, `system-test`, `samples-test`). +* `tsconfig.json`: Configures the TypeScript compilation options. +* `.mocharc.js`: Specifies test execution configurations for Mocha (timeouts, reporter settings, etc.). +* `.nycrc` / `c8`: Configuration settings for tracking code coverage percentages. +* `.eslintrc.js` / `gts`: Standard Google styling and lint configurations. +* `.jsdoc.js`: Configuration settings for compiling API documentation. diff --git a/core/packages/google-auth-library-nodejs/CODEMETA.md b/core/packages/google-auth-library-nodejs/CODEMETA.md new file mode 100644 index 000000000000..52d52395e160 --- /dev/null +++ b/core/packages/google-auth-library-nodejs/CODEMETA.md @@ -0,0 +1,369 @@ +# Google Auth Library for Node.js: Architectural & File Reference +# *WARNING*: This file is AI generated and may contain inaccuracies. + +Welcome to the codebase metadata directory reference for the `@google-cloud/auth-library` Node.js client. This document serves as a deep-dive technical onboarding guide and reference, detailing the architecture, flow, inner workings, and test suites of this library. + +--- + +## Table of Contents +1. [Package Purpose & High-Level Architecture](#1-package-purpose--high-level-architecture) +2. [Core Architectural Design Patterns](#2-core-architectural-design-patterns) +3. [A Tour of a Token Request (Execution Flow)](#3-a-tour-of-a-token-request-execution-flow) +4. [Deep Dive: Source Directory Mechanics (`src/`)](#4-deep-dive-source-directory-mechanics-src) + - [Root Utilities (`src/util.ts`)](#root-utilities-srcutilts) + - [Cryptographic Adapter Layer (`src/crypto/`)](#cryptographic-adapter-layer-srccrypto) + - [Service Account Token Engine (`src/gtoken/`)](#service-account-token-engine-srcgtoken) + - [Identity Federation & STS Layer (`src/auth/`)](#identity-federation--sts-layer-srcauth) +5. [Exhaustive File-by-File Reference](#5-exhaustive-file-by-file-reference) + - [Root & Core Source Files](#root--core-source-files) + - [Authentication Clients & Identity Federation (`src/auth/`)](#authentication-clients--identity-federation-srcauth-1) + - [Cryptographic Interfaces & Implementations (`src/crypto/`)](#cryptographic-interfaces--implementations-srccrypto-1) + - [Token Caching & Fetching Submodule (`src/gtoken/`)](#token-caching--fetching-submodule-srcgtoken-1) +6. [Testing Mechanics & Test Suites (`test/`)](#6-testing-mechanics--test-suites-test) + - [Testing Design Patterns](#testing-design-patterns) + - [Core Library Unit Tests](#core-library-unit-tests) + - [gtoken Submodule Unit Tests](#gtoken-submodule-unit-tests) + - [System & Integration Tests (`system-test/`)](#system--integration-tests-system-test) + - [Browser Tests (`browser-test/`)](#browser-tests-browser-test) + +--- + +## 1. Package Purpose & High-Level Architecture + +The `google-auth-library-nodejs` package is the shared authentication layer powering all Google Cloud Platform Node.js clients (e.g., `@google-cloud/storage`, `@google-cloud/pubsub`, and GAPIC-generated libraries). It manages **OAuth 2.0 user consent loops**, **JSON Web Tokens (JWT)**, **Service Accounts**, **Google Compute Engine Serverless Metadata**, and **Identity Federation (Workload & Workforce)**. + +The following diagram shows how the central coordinator class (`GoogleAuth`) resolves environmental factors and configures `AuthClient` subclasses: + +```mermaid +graph TD + A[Application Code] --> B(GoogleAuth Coordinator) + B --> C{Detect Environment & Config} + + C -->|Local JSON Keyfile| D[JWT Client] + C -->|GCP Server / GCE / GKE / Cloud Run| E[Compute Client] + C -->|AWS Workload Identity Config| F[AwsClient] + C -->|Identity Pool Config K8s/Azure/mTLS| G[IdentityPoolClient] + C -->|Pluggable External Script| H[PluggableAuthClient] + C -->|OAuth User Consent Code| I[OAuth2Client] + + D & E & F & G & H & I -->|Extends / Implements| J(AuthClient Base) + J -->|Wraps HTTP Request| K[Gaxios Transporter] + K -->|1. Exchange Credentials| L[GCP STS / OAuth 2.0 Endpoints] + L -->|2. Return Access Token| K + K -->|3. Inject Authorization Bearer Header| M[Secure Google API Endpoint] +``` + +--- + +## 2. Core Architectural Design Patterns + +The library relies on four key software design patterns: + +* **The Transporter Adapter Pattern**: The abstract `AuthClient` class leverages `Gaxios` as its network transporter. It establishes request/response interceptors that inject telemetry, log debugging metrics, and implement exponential backoff retries for temporary network issues. +* **Dynamic Platform Cryptographic Abstraction**: Bridges standard Node.js server runtimes with web browser environments by exposing a single unified `Crypto` interface. An abstract factory (`crypto.ts`) checks environment constraints at startup and swaps between Node's native C++ bindings (`NodeCrypto`) and standard browser-based asynchronous Web Crypto (`BrowserCrypto`). +* **The Supplier Pattern**: Separates discovery of credentials from the token exchange flow. For instance, `SubjectTokenSupplier` abstracts how mTLS certificates, AWS security credentials, or pluggable binary STDOUT values are resolved, making the core exchange logic generic. +* **Environment Discovery Engine**: Pre-configures Application Default Credentials (ADC) using sequential checks across environment variables (`GOOGLE_APPLICATION_CREDENTIALS`), gcloud configuration profiles, and the VM metadata server. + +--- + +## 3. A Tour of a Token Request (Execution Flow) + +When an application makes a request to a Google API (e.g., uploading a file to Google Cloud Storage), the library executes the following multi-step sequence under the hood: + +### Step 1: Client Selection (The ADC Discovery Phase) +When the user invokes `new GoogleAuth().getClient()`, the library triggers its discovery sequence in `googleauth.ts`: +1. **Environment Variables**: Checks `process.env.GOOGLE_APPLICATION_CREDENTIALS`. If set, it reads the file and invokes `_getApplicationCredentialsFromFilePath`. + - If the file contains `type: "service_account"`, it instantiates a `JWT` client. + - If the file contains `type: "external_account"`, it hands the payload to `ExternalAccountClient.fromJSON()`, which returns an `AwsClient`, `IdentityPoolClient`, or `PluggableAuthClient`. +2. **Well-Known Developer Path**: If the environment variable is absent, it looks for a JSON file created by `gcloud auth application-default login`: + - Linux/Mac: `~/.config/gcloud/application_default_credentials.json` + - Windows: `%APPDATA%/gcloud/application_default_credentials.json` +3. **Metadata Server**: If no credential file is found, it checks whether it's running on GCP (using the metadata server). If the server is reachable, it instantiates a `Compute` client. +4. **Fail-Safe**: If none of these are found, it throws a `NO_ADC_FOUND` exception. + +``` +[Application] + β”‚ + β–Ό +new GoogleAuth().getClient() + β”‚ + β”œβ”€β”€β–Ί 1. Env Var GOOGLE_APPLICATION_CREDENTIALS? ──► Parse file and return client + β”‚ + β”œβ”€β”€β–Ί 2. Well-Known Local Developer Credentials? ──► Parse file and return client + β”‚ + β”œβ”€β”€β–Ί 3. Running on GCP (Metadata Server)? ────────► Instantiate and return Compute Client + β”‚ + └──► 4. None? ──────────────────────────────────► Throw "ADC not found" error +``` + +### Step 2: Request Interception & Telemetry Injection +All requests to Google APIs are routed through `client.request(opts)`. The base `AuthClient` handles request interception: +- Automatically resolves and caches the target domain (e.g. checking if we are executing in the standard `googleapis.com` or custom private VPC networks). +- Injects standard Google telemetry headers (`x-goog-api-client` containing runtime and SDK versions, alongside standard billing project overrides `x-goog-user-project`). + +### Step 3: Token Retrieval or Exchange Phase +If no valid access token is cached (or the cached token has expired or is expiring soon), the client triggers its internal token retrieval loop: +* **Service Account / JWT (`jwtclient.ts` / `gtoken`)**: + 1. Constructs a JSON Web Signature (JWS) claim containing the service account email, requested scopes, audience, and expiry timestamp. + 2. Signs the claim with the service account's private key using the **RS256** algorithm (delegated to the `Crypto` interface). + 3. Sends the signed assertion as an HTTP POST request to `oauth2.googleapis.com/token` to retrieve a fresh OAuth 2.0 access token. +* **Workload Identity Federation (`baseexternalclient.ts` / `stscredentials.ts`)**: + 1. The client resolves a third-party "subject token" (e.g., a Kubernetes Service Account token, an OIDC ID token, an mTLS cert, or an AWS signature). + 2. It bundles this subject token into a standard RFC 8693 Token Exchange request payload (`grant_type=urn:ietf:params:oauth:grant-type:token-exchange`). + 3. Issues an HTTP call to `sts.googleapis.com` to exchange the third-party token for a federated GCP access token. + 4. If a `service_account_impersonation_url` is provided in the configuration, it takes the federated access token and calls the Google `iamcredentials` API to impersonate a target service account, returning the final impersonated token to the caller. +* **Metadata Server (`computeclient.ts`)**: + 1. Queries the local GCE/GKE metadata server at `http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token`. + 2. Appends the required `Metadata-Flavor: Google` header. + 3. Parses the returned access token from the JSON response. + +### Step 4: Safe Request Replay on Failure +If a request fails due to authentication issues (returning a `401 Unauthorized` or `403 Forbidden`), the `requestAsync` interceptor checks if a token refresh has already been attempted. If not, it clears the cached credentials, requests a new access token, and replays the request once. + +--- + +## 4. Deep Dive: Source Directory Mechanics (`src/`) + +### Root Utilities (`src/util.ts`) + +#### Normalizing Casing +Configurations provided in JSON files often use `snake_case` (compliant with standard OAuth/Google configurations), while TypeScript classes prefer `camelCase`. `util.ts` provides a custom helper, `originalOrCamelOptions`, which wraps options in a `Map` structure to support lookups using both formats: +```typescript +const opts = originalOrCamelOptions(options); +const tokenUrl = opts.get('token_url'); // will match 'token_url' or 'tokenUrl' +``` + +#### The Cache System (`LRUCache`) +To avoid redundant token fetches, the library maintains a basic memory-bound Least Recently Used (LRU) cache. Cached objects are automatically pruned when the cache reaches its maximum capacity, ensuring the library uses minimal memory even during high-throughput credential flows. + +--- + +### Cryptographic Adapter Layer (`src/crypto/`) + +The cryptographic layer dynamically routes signature and hashing operations depending on the execution environment. + +``` + [createCrypto() Factory] + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β” + β–Ό β–Ό + [Browser Environment] [Node.js Environment] + β”‚ β”‚ + β–Ό β–Ό + BrowserCrypto NodeCrypto + (Web Crypto API) (Native crypto) +``` + +* **`NodeCrypto`**: Uses the native C++ Node.js `crypto` bindings. Operations such as `verify` and `sign` run synchronously under the hood, but are wrapped in Promises to maintain interface compatibility. +* **`BrowserCrypto`**: Uses the browser's asynchronous `window.crypto.subtle` interface. + - Since standard PEM private keys are not supported directly by Web Crypto, public key verification is performed by importing certificates formatted as **JSON Web Keys (JWK)**. + - It uses `base64-js` and native `TextEncoder`/`TextDecoder` structures to handle string/byte array mappings across environments. + +--- + +### Service Account Token Engine (`src/gtoken/`) + +`gtoken` is a self-contained token manager focused on service account authorization. It does not inherit from `AuthClient`, making it ideal for lightweight utility scripts or microservices. + +* **Preventing Thundering Herds (`tokenHandler.ts`)**: When an access token expires, concurrent API calls could trigger multiple parallel network requests to fetch a new token. `TokenHandler` solves this by caching the in-flight promise (`#pendingToken`). Subsequent callers share the same promise, ensuring only one network request is made: + ```typescript + // tokenHandler.ts conceptual structure + async getToken(forceRefresh: boolean): Promise { + if (this.token && !this.hasExpired() && !forceRefresh) { + return this.token; + } + if (this.pendingPromise) { + return this.pendingPromise; + } + this.pendingPromise = this.fetchToken(); + try { + this.token = await this.pendingPromise; + return this.token; + } finally { + this.pendingPromise = null; + } + } + ``` +* **JWS Signature Creation (`jwsSign.ts`)**: Builds a JWT assertion by base64url-encoding the header (`{"alg":"RS256","typ":"JWT"}`) and claims, concatenating them with a dot (`.`), and signing the resulting string using the private key and **RS256**. + +--- + +### Identity Federation & STS Layer (`src/auth/`) + +Identity Federation allows workloads running outside GCP (e.g., AWS, Azure, Kubernetes) to authenticate to Google APIs without using long-lived Service Account private keys. + +#### Base External Client & RFC 8693 Exchange +`BaseExternalAccountClient` coordinates this flow using the GCP Security Token Service (STS): +1. Invokes `retrieveSubjectToken()` (which is implemented by subclasses depending on the environment). +2. Sends an HTTP POST request to `sts.googleapis.com/v1/token` with the following parameters: + - `grant_type`: `urn:ietf:params:oauth:grant-type:token-exchange` + - `subject_token`: The resolved identity token. + - `subject_token_type`: The type of token (e.g. `urn:ietf:params:oauth:token-type:jwt` or `urn:ietf:params:aws:token-type:aws4_request`). + - `audience`: The fully specified resource path of the workload pool provider. + +#### Pluggable Auth Script Executor +For custom hosting environments, users can configure a pluggable executable script. The library spawns this script using `child_process` via `PluggableAuthHandler`: +* **Execution Safety**: + - Enforces configurable execution timeouts to prevent blocking the application process. + - Restricts the size of the output read from `stdout` (capping it at 64 KB) to protect against memory exhaustion or Denial of Service (DoS) issues. + - Validates the script's standard output against a structured JSON schema (`ExecutableResponse`) before passing it to the STS exchange layer. + +--- + +## 5. Exhaustive File-by-File Reference + +### Root & Core Source Files + +| File | Purpose | Working Mechanics & Structure | +| :--- | :--- | :--- | +| [`index.ts`](src/index.ts) | Public Exports | Re-exports the entire public API surface. Serves as a facade, allowing consumers to import all key classes directly from a single module. | +| [`shared.cts`](src/shared.cts) | Telemetry & Headers | Reads the library version from `package.json` at initialization and exports the standard `USER_AGENT` header template used by Google APIs for telemetry. | +| [`util.ts`](src/util.ts) | Common Utilities | Contains casing normalizers, the memory-cached `LRUCache` implementation, and helper functions to validate local certificate paths. | + +### Authentication Clients & Identity Federation (`src/auth/`) + +| File | Purpose | Working Mechanics & Structure | +| :--- | :--- | :--- | +| [`authclient.ts`](src/auth/authclient.ts) | Abstract Base Client | Defines the core `AuthClient` class. Implements the default HTTP `request` wrapper, handles retry logic, and manages standard headers (such as `x-goog-user-project` and universe domain verification). | +| [`googleauth.ts`](src/auth/googleauth.ts) | ADC Coordinator | Orchestrates Application Default Credentials (ADC) discovery. Traverses environment variables, local configuration directories, and the metadata server to instantiate the appropriate client. | +| [`credentials.ts`](src/auth/credentials.ts) | Interfaces & Models | Defines core TypeScript interfaces representing access tokens, refresh tokens, service account keys, and external configuration files. | +| [`envDetect.ts`](src/auth/envDetect.ts) | Environment Detection | Checks system environment variables and queries metadata endpoints to determine if the library is running on GCE, GKE, App Engine, or serverless environments. | +| [`baseexternalclient.ts`](src/auth/baseexternalclient.ts) | Identity Federation Base | Coordinates RFC 8693 token exchanges and handles subsequent Service Account impersonation calls via Google's `iamcredentials` API. | +| [`awsclient.ts`](src/auth/awsclient.ts) | AWS Identity Client | Implements `BaseExternalAccountClient` for AWS. Resolves temporary AWS security credentials and generates signed STS requests. | +| [`awsrequestsigner.ts`](src/auth/awsrequestsigner.ts) | AWS Signature V4 Signer | Signs HTTP requests directed to AWS services using the AWS Signature Version 4 signing algorithm, generating standard `Authorization` and `x-amz-date` headers. | +| [`defaultawssecuritycredentialssupplier.ts`](src/auth/defaultawssecuritycredentialssupplier.ts) | AWS Credentials Resolver | Resolves AWS IAM temporary credentials by querying local environment variables or calling the local AWS EC2 instance metadata service. | +| [`identitypoolclient.ts`](src/auth/identitypoolclient.ts) | Workload Identity Client | Implements `BaseExternalAccountClient` for standard OIDC identity pools. Leverages suppliers to load subject tokens from files, URLs, or certificates. | +| [`filesubjecttokensupplier.ts`](src/auth/filesubjecttokensupplier.ts) | File Token Supplier | Implements `SubjectTokenSupplier` to read subject tokens from local text or JSON files (such as Kubernetes projected volumes). | +| [`urlsubjecttokensupplier.ts`](src/auth/urlsubjecttokensupplier.ts) | URL Token Supplier | Implements `SubjectTokenSupplier` to retrieve subject tokens via HTTP GET requests from local service endpoints. | +| [`certificatesubjecttokensupplier.ts`](src/auth/certificatesubjecttokensupplier.ts) | Certificate Token Supplier | Implements `SubjectTokenSupplier` to read and format base64 X.509 client certificates for Mutual TLS (mTLS) flows. | +| [`oauth2client.ts`](src/auth/oauth2client.ts) | OAuth 2.0 User Client | Manages user authorization code loops, exchanges codes for tokens, verifies PKCE challenges, and validates ID token signatures. | +| [`oauth2common.ts`](src/auth/oauth2common.ts) | OAuth Utilities | Provides shared helpers to inject client credentials, format request bodies, and parse standard OAuth HTTP error responses. | +| [`jwtclient.ts`](src/auth/jwtclient.ts) | Service Account JWT Client | Manages Service Account token flows by building self-signed JWT assertions and exchanging them with Google's token endpoint. | +| [`jwtaccess.ts`](src/auth/jwtaccess.ts) | Self-Signed JWT Bearer | Generates signed JWTs that are injected directly as Bearer tokens in outgoing HTTP requests, bypassing the need to exchange them for an access token. | +| [`computeclient.ts`](src/auth/computeclient.ts) | GCE Metadata Client | Queries the local GCP metadata server to retrieve and refresh access tokens for GCE instances and GKE nodes. | +| [`downscopedclient.ts`](src/auth/downscopedclient.ts) | Credential Access Boundary | Interacts with Google's STS to exchange a standard credential for a restricted access token with scoped permissions. | +| [`executable-response.ts`](src/auth/executable-response.ts) | External Script Response | Parses and validates the JSON response returned by pluggable authentication scripts. | +| [`pluggable-auth-client.ts`](src/auth/pluggable-auth-client.ts) | Pluggable Script Client | Implements `BaseExternalAccountClient`. Executes external scripts to retrieve subject tokens and exchanges them with GCP's STS. | +| [`pluggable-auth-handler.ts`](src/auth/pluggable-auth-handler.ts) | Pluggable Process Handler | Spawns child processes to execute pluggable scripts, enforcing timeouts and buffer size limits. | +| [`externalAccountAuthorizedUserClient.ts`](src/auth/externalAccountAuthorizedUserClient.ts) | Federated User Client | Manages long-lived refresh tokens for workforce-federated users, coordinating token exchanges and rotations. | +| [`externalclient.ts`](src/auth/externalclient.ts) | External Client Factory | A static factory that parses external credential configurations and instantiates the correct client subclass. | +| [`idtokenclient.ts`](src/auth/idtokenclient.ts) | ID Token Client | Retrieves OpenID Connect (OIDC) ID tokens from the metadata server or exchanges credentials for them, primarily for service-to-service authentication. | +| [`impersonated.ts`](src/auth/impersonated.ts) | Impersonation Client | Impersonates a target Service Account by exchanging a source credential for an impersonated token via the IAM credentials API. | +| [`iam.ts`](src/auth/iam.ts) | Legacy IAM Headers | Implements a simple wrapper to manually inject Cloud IAM authority headers into outgoing requests. | +| [`loginticket.ts`](src/auth/loginticket.ts) | Decoded OIDC ID Ticket | Parses OIDC ID tokens and exposes their claims (such as user ID, email, and audience) via getter methods. | +| [`refreshclient.ts`](src/auth/refreshclient.ts) | User Refresh Client | Refreshes OAuth 2.0 user credentials using standard long-lived OAuth refresh tokens. | +| [`stscredentials.ts`](src/auth/stscredentials.ts) | STS Token Client | Communicates with Google's Security Token Service (STS) to exchange third-party tokens for GCP access tokens. | +| [`passthrough.ts`](src/auth/passthrough.ts) | No-op Client | A no-op client that forwards requests without adding authentication headers. Useful when connecting to local emulators. | + +### Cryptographic Interfaces & Implementations (`src/crypto/`) + +| File | Purpose | Working Mechanics & Structure | +| :--- | :--- | :--- | +| [`shared.ts`](src/crypto/shared.ts) | Cryptographic Interface | Defines common cryptographic type signatures and interfaces (hashing, signing, verification) used throughout the library. | +| [`crypto.ts`](src/crypto/crypto.ts) | Cryptographic Factory | Exposes the `createCrypto` factory function, which dynamically instantiates `NodeCrypto` or `BrowserCrypto` based on runtime environment checks. | +| [`node/crypto.ts`](src/crypto/node/crypto.ts) | Node.js Crypto Provider | Implements the `Crypto` interface using native Node.js C++ bindings. Signs assertions and validates certificates. | +| [`browser/crypto.ts`](src/crypto/browser/crypto.ts) | Browser Crypto Provider | Implements the `Crypto` interface using browser Web Crypto (`window.crypto.subtle`) and JSON Web Keys (JWK). | + +### Token Caching & Fetching Submodule (`src/gtoken/`) + +| File | Purpose | Working Mechanics & Structure | +| :--- | :--- | :--- | +| [`googleToken.ts`](src/gtoken/googleToken.ts) | GToken Entry Point | The public API of the `gtoken` submodule. Wraps `TokenHandler` and exposes simplified token acquisition and revocation methods. | +| [`tokenOptions.ts`](src/gtoken/tokenOptions.ts) | GToken Options | Defines configuration properties for Service Accounts, including private keys, scopes, and transport settings. | +| [`tokenHandler.ts`](src/gtoken/tokenHandler.ts) | Token Handler | Coordinates in-flight promises and caches tokens to prevent parallel token requests (thundering herds). | +| [`getCredentials.ts`](src/gtoken/getCredentials.ts) | Key File Resolver | Resolves private keys from raw configuration files (supporting `.pem`, `.json`, and `.p12` formats). | +| [`jwsSign.ts`](src/gtoken/jwsSign.ts) | JWS Assertion Signer | Signs a base64 JWS assertion with a private key using the RS256 algorithm. | +| [`getToken.ts`](src/gtoken/getToken.ts) | Token HTTP Client | Sends signed JWT assertions as HTTP POST requests to Google's authorization servers. | +| [`revokeToken.ts`](src/gtoken/revokeToken.ts) | Token Revocation | Sends an HTTP request to Google's revocation endpoint to invalidate active tokens. | +| [`errorWithCode.ts`](src/gtoken/errorWithCode.ts) | Custom Error | Defines a custom error class that bundles descriptive messages with alphanumeric error codes. | + +--- + +## 6. Testing Mechanics & Test Suites (`test/`) + +### Testing Design Patterns + +* **Fake Timers Sandbox**: Token expiration logic is time-sensitive. Using real timeouts in tests would cause delays and slow down the suite. The tests use `sinon.useFakeTimers` to simulate the passage of time, allowing them to verify expiration and refresh logic instantly: + ```typescript + const clock = sinon.useFakeTimers(); + // ... initiate cached token ... + clock.tick(3600 * 1000); // fast-forward 1 hour + assert.ok(client.isTokenExpiring()); + clock.restore(); + ``` +* **HTTP Network Mocking (`nock`)**: To prevent the tests from making real network requests to Google API endpoints, the suite uses `nock` to intercept and mock HTTP calls. This allows the tests to assert that the correct request payloads, headers, and query parameters are sent: + ```typescript + nock('https://oauth2.googleapis.com') + .post('/token') + .reply(200, { + access_token: 'mock-access-token', + expires_in: 3600, + }); + ``` + +### Core Library Unit Tests + +| File | Targets & Test Coverage | +| :--- | :--- | +| [`test.authclient.ts`](test/test.authclient.ts) | Verifies `AuthClient` default properties, request interceptors, quota project propagation, and method-level call logging. | +| [`test.googleauth.ts`](test/test.googleauth.ts) | Tests the ADC discovery chain, verifying correct resolution order across environment paths and well-known local files. | +| [`test.oauth2.ts`](test/test.oauth2.ts) | Tests user consent loops, authorization code exchanges, PKCE verifications, refresh loops, and event emissions. | +| [`test.oauth2common.ts`](test/test.oauth2common.ts) | Asserts base64 URL formatting, error payload mapping, and common header injection logic. | +| [`test.jwt.ts`](test/test.jwt.ts) | Verifies the `JWT` client, checking key file parsing, RS256 signature generation, and token caching behaviors. | +| [`test.jwtaccess.ts`](test/test.jwtaccess.ts) | Tests self-signed JWT bearer token generation, asserting that correct claims and audiences are signed. | +| [`test.compute.ts`](test/test.compute.ts) | Mocks the GCE metadata server to test token retrieval, ID token fetching, and error wrapping for 403/404 responses. | +| [`test.downscopedclient.ts`](test/test.downscopedclient.ts) | Asserts that the `DownscopedClient` correctly serializes Credential Access Boundary (CAB) rules during token exchanges. | +| [`test.baseexternalclient.ts`](test/test.baseexternalclient.ts) | Verifies the base identity federation logic, including STS exchanges, process retries, and concurrent request caching. | +| [`test.awsclient.ts`](test/test.awsclient.ts) | Mocks AWS IMDSv2 endpoints to test regional STS exchanges and token exchanges. | +| [`test.awsrequestsigner.ts`](test/test.awsrequestsigner.ts) | Compares calculated AWS Signature Version 4 hashes and query parameters against expected AWS test suites. | +| [`test.identitypoolclient.ts`](test/test.identitypoolclient.ts) | Verifies identity pool configurations, testing subject token retrieval from files, URLs, and certificates. | +| [`test.executableresponse.ts`](test/test.executableresponse.ts) | Asserts that the library correctly handles malformed JSON or missing fields in pluggable executable output. | +| [`test.pluggableauthclient.ts`](test/test.pluggableauthclient.ts) | Mocks pluggable commands and verifies that the returned subject token is exchanged with the GCP STS endpoint. | +| [`test.pluggableauthhandler.ts`](test/test.pluggableauthhandler.ts) | Verifies command execution, checking that process timeouts, buffer size limits, and error outputs are handled correctly. | +| [`test.externalclient.ts`](test/test.externalclient.ts) | Exercises the `ExternalAccountClient` factory to confirm it instantiates the correct client subclass from JSON configurations. | +| [`test.externalaccountauthorizeduserclient.ts`](test/test.externalaccountauthorizeduserclient.ts) | Tests federated user token rotations and refresh token updates. | +| [`test.idtokenclient.ts`](test/test.idtokenclient.ts) | Verifies that serverless service-to-service ID tokens are retrieved and refreshed correctly. | +| [`test.impersonated.ts`](test/test.impersonated.ts) | Tests Service Account impersonation, verifying that the library requests tokens with the correct scopes and lifetime parameters. | +| [`test.refresh.ts`](test/test.refresh.ts) | Tests user credential refreshing using mock OAuth refresh token exchanges. | +| [`test.stscredentials.ts`](test/test.stscredentials.ts) | Verifies that raw STS token exchange requests are constructed and serialized correctly. | +| [`test.passthroughclient.ts`](test/test.passthroughclient.ts) | Confirms that the no-op client forwards request headers without modifying them. | +| [`test.crypto.ts`](test/test.crypto.ts) | Verifies cryptographic operations (hashing, signing, and verification) using the `NodeCrypto` provider. | +| [`test.loginticket.ts`](test/test.loginticket.ts) | Checks that decoded login ticket claims are exposed correctly via getter methods. | +| [`test.iam.ts`](test/test.iam.ts) | Verifies that the legacy IAM client correctly injects headers into outgoing requests. | +| [`test.util.ts`](test/test.util.ts) | Tests core utilities, including LRU caching and casing conversions. | +| [`test.index.ts`](test/test.index.ts) | Verifies that all public API surfaces are exported correctly from the library's entry point. | + +### gtoken Submodule Unit Tests + +| File | Targets & Test Coverage | +| :--- | :--- | +| [`test.googleToken.ts`](test/gtoken/test.googleToken.ts) | Verifies GToken token acquisition, caching, and revocation flows. | +| [`test.tokenHandler.ts`](test/gtoken/test.tokenHandler.ts) | Tests eager token refreshes, clock offsets, and concurrent promise caching. | +| [`test.getCredentials.ts`](test/gtoken/test.getCredentials.ts) | Verifies key file parsing for `.json`, `.pem`, and `.p12` formats. | +| [`test.jwsSign.ts`](test/gtoken/test.jwsSign.ts) | Asserts that RS256 JWT assertions are signed and formatted correctly. | +| [`test.getToken.ts`](test/gtoken/test.getToken.ts) | Mocks Google's token servers to verify response error handling. | +| [`test.revokeToken.ts`](test/gtoken/test.revokeToken.ts) | Verifies that token revocation requests are sent correctly. | +| [`test.errorWithCode.ts`](test/gtoken/test.errorWithCode.ts) | Ensures custom errors expose error codes alongside standard descriptive messages. | + +--- + +## 5. System & Integration Tests (`system-test/`) + +System tests verify package compiling and live environment execution. + +* **[`test.kitchen.ts`](system-test/test.kitchen.ts)**: + - **How it works**: Creates a temporary testing environment and bundles the library using standard production package configurations. + - **What it asserts**: Verifies that the bundled library builds and compiles under standard consumer TypeScript configurations, checking that all exported types resolve correctly without runtime errors. + +--- + +## 6. Browser Tests (`browser-test/`) + +Browser tests run inside a headless browser environment to verify cross-platform compatibility. + +* **[`test.crypto.ts`](browser-test/test.crypto.ts)**: + - **How it works**: Executes cryptographic operations inside the browser runtime. + - **What it asserts**: Verifies that `BrowserCrypto` (using the browser's Web Crypto API) successfully performs SHA-256 hashing and RS256 signatures, producing identical results to Node's native C++ crypto implementation. +* **[`test.oauth2.ts`](browser-test/test.oauth2.ts)**: + - **How it works**: Simulates OAuth flows inside the browser runtime. + - **What it asserts**: Verifies that PKCE verifiers, state challenges, and authorization URL configurations compile and run smoothly in browser-based applications. diff --git a/core/packages/logging-utils/CODEMETA.md b/core/packages/logging-utils/CODEMETA.md new file mode 100644 index 000000000000..620a477b1d06 --- /dev/null +++ b/core/packages/logging-utils/CODEMETA.md @@ -0,0 +1,212 @@ +# google-logging-utils Architecture Guide +# *WARNING*: This file is AI generated and may contain inaccuracies. + +Welcome to the architectural reference and code tour for `google-logging-utils` (npm package: `google-logging-utils`). + +This package is a lightweight, highly-optimized, zero-dependency (excluding dev dependencies) logging utility used internally by other Google Cloud Node.js client libraries (such as `@google-cloud/pubsub`) to manage diagnostic, debug, and structured trace logging. + +--- + +## 🎯 Core Purpose & Design Philosophy + +The core philosophy of `google-logging-utils` is to provide a flexible and powerful ad-hoc diagnostic logging layer that is **completely unobtrusive** in production and **extremely rich** when enabled. + +Key architectural goals include: +1. **Zero Runtime Overhead when Disabled**: If diagnostic logging is not explicitly enabled via environment variables or manual configuration, calling `log()` resolves to a static, pre-instantiated placeholder function with zero memory or CPU overhead. No loggers are allocated in memory. +2. **Event-Driven Observability**: Every active logger instance acts as an `EventEmitter`. This allows libraries or host applications to dynamically "tap" into diagnostic logs by listening to the `'log'` event, facilitating custom ingestion or local diagnostic streams. +3. **Dynamic Hot-Swapping**: Loggers are decoupled from their actual stdout/stderr emitters (backends). Developers can hot-swap backends globally at runtime (e.g., switching from plain text to Google Cloud Structured JSON logging) without losing previously registered event listeners. +4. **Zero Production Dependencies**: To minimize the package foot-print and prevent dependency conflicts in user applications, all terminal colorization and temporal helpers are built from scratch without pulling in external libraries. + +--- + +## 🧱 System Architecture & Design Patterns + +### 1. The Dynamic Backend Bridge +```mermaid +sequenceDiagram + autonumber + participant App as Host Application + participant LogFn as AdhocDebugLogFunction (func) + participant LogInst as AdhocDebugLogger (instance) + participant Mgr as Logging Manager / Cache + participant Backend as DebugLogBackend (e.g., NodeBackend) + + App->>Mgr: log('my-system') + Note over Mgr: Check env / backend state + Mgr->>LogInst: Create AdhocDebugLogger + LogInst-->>LogFn: Return wrapped callable (func) + Mgr->>Mgr: Cache logger by namespace + Mgr-->>App: Return func + + rect rgb(240, 245, 255) + Note over App, Backend: Dynamic Log Invocation Flow + App->>LogFn: info('Something happened') + LogFn->>LogInst: invokeSeverity('INFO', 'Something happened') + LogInst->>Backend: log('my-system', {severity: 'INFO'}, 'Something happened') + Backend->>Backend: Format & Output (console / stderr) + LogInst->>LogInst: emit('log', {severity: 'INFO'}, ['Something happened']) + end +``` + +### 2. Architecture Patterns in Action + +#### The Placeholder Pattern (Zero-Overhead Fallback) +If neither the global `cachedBackend` nor the environment variable `GOOGLE_SDK_NODE_LOGGING` are present, `log()` immediately returns a pre-created, no-op logger `placeholder` instead of compiling regexes, reading environment flags, or allocating a new class. + +#### The Self-Referential Function Pattern +An `AdhocDebugLogFunction` returned by `log()` is both a **callable function** (for simple logs) and an **object/class** that exposes methods and events. +```typescript +const debug = log('pubsub'); + +// Callable style (default severity) +debug({other: {foo: 'bar'}}, 'Initiating connection'); + +// Severity method style +debug.info('Connected!'); +debug.error('Failed to connect'); + +// Event emitter style +debug.on('log', (fields, args) => { + console.log('Tapped diagnostic line:', args); +}); +``` + +#### The Sub-Logger Pattern +Using `debug.sublog('connection')` creates a child logger. The system automatically propagates namespaces hierarchically by concatenating them with a colon (`pubsub:connection`), which allows fine-grained regex filtering. + +--- + +## πŸ“‚ Directory & File Tour + +Here is an overview of all key files within `core/packages/logging-utils`: + +| File Path | Role | Core Responsibility | +| :--- | :--- | :--- | +| **`src/index.ts`** | Entrypoint | Defines public exports, including all logging factory helpers and interfaces. | +| **`src/types.ts`** | Domain Types | Declares the core enums, types, and functional interfaces representing log fields, severities, and pluggable backends. | +| **`src/colours.ts`** | Terminal Styling | Implements a robust, dependency-free ANSI terminal color detector and colorizer inspired by Node's internal utility. | +| **`src/temporal.ts`** | Time Utils | Implements a simplified, TC39-compliant shim for managing durations and milliseconds without external dependencies. | +| **`src/logging-utils.ts`** | Logging Orchestration | The heartbeat of the library. Manages namespaces, registry caching, environment filtering, and pluggable backend implementations. | +| **`test/logging-utils.ts`** | Unit Tests | Validates logging behavior, filter regex parsing, structured JSON formatting, event tapping, and caching. | +| **`test/temporal.ts`** | Unit Tests | Assures correctness of the TC39 `Duration` polyfill across multiple time units. | +| **`samples/system-test/test.quickstart.ts`** | Integration Test | Runs quickstart samples inside isolated child processes to ensure environment flag routing works. | + +--- + +## πŸ” Detailed Module Walkthrough + +### 1. `src/index.ts` +A simple entrypoint that re-exports all symbols from `logging-utils` and exposes selected types from `types.ts`. + +--- + +### 2. `src/types.ts` +Declares the core data contracts for the library: +- **`LogSeverity`**: An enum defining GCP-compatible log severities (`DEFAULT`, `DEBUG`, `INFO`, `WARNING`, `ERROR`). +- **`LogFields`**: The metadata envelope that can contain OpenTelemetry details (`telemetryTraceId`, `telemetrySpanId`), `severity`, and a catch-all `other` field for custom structured metadata. +- **`AdhocDebugLogFunction`**: The hybrid interface returned to users. It can be directly invoked as a function, has methods for severities (`.info()`, `.warn()`, etc.), manages child namespaces (`.sublog()`), and implements standard `EventEmitter` `.on('log')` subscriptions. +- **`DebugLogBackend`**: The interface that pluggable log targets must implement, consisting of `log()` and `setFilters()`. + +--- + +### 3. `src/colours.ts` +Terminal styling is critical for developer logs, but importing third-party color libraries (`colors`, `chalk`, etc.) bloats dependencies. `colours.ts` implements a localized `Colours` helper: +- **TTY & Capability Detection**: Uses `tty.WriteStream.isTTY` and `getColorDepth()` to dynamically check if terminal outputs support colourization. +- **Fallback Handling**: If the target stream is not a TTY or has insufficient color depth, colors are disabled by substituting ANSI sequences with empty strings (`''`). +- **ANSI Sequence Mapping**: Provides standard escape codes for terminal color styling (`red`, `green`, `yellow`, `blue`, `magenta`, `cyan`, `white`, `grey`, `dim`, `bright`, `reset`). + +--- + +### 4. `src/temporal.ts` +This file bridges the gap until the **TC39 Temporal** specification becomes native across all supported Node.js runtimes. +- **`Duration` Class**: An immutable class wrapping a private `millis` value. +- **`Duration.from(DurationLike)`**: Safely builds a duration from objects containing fractional hours, minutes, seconds, or milliseconds (e.g., `{ minutes: 30 }`). +- **`totalOf(TotalOfUnit)`**: Converts a stored duration back into a floating-point representation of a requested unit (e.g. converting a 90-minute duration to `1.5` hours or `90` minutes). + +> [!NOTE] +> Since polyfills are often heavy or contain runtime compatibility bugs on legacy Node versions, this highly-simplified shim provides only the exact subset of `Temporal.Duration` required by client libraries, keeping the runtime bundle size small. + +--- + +### 5. `src/logging-utils.ts` +The primary core module implementing all logical operations: + +#### `AdhocDebugLogger` +A class extending `EventEmitter` that binds log invocations to dynamic targets. It sets up convenience methods (`.debug`, `.info`, `.warn`, `.error`) and handles exception safety. If an upstream backend or a custom event subscriber throws an error during execution, the exception is caught and swallowed to guarantee logging never disrupts the host application. + +#### `DebugLogBackendBase` +The abstract base class that manages namespace filters. It parses the filter configuration string provided in the `GOOGLE_SDK_NODE_LOGGING` environment variable (comma-separated namespaces, allowing wildcard `*` symbols). + +#### Pluggable Backends +1. **`NodeBackend`**: The default, highly-optimized text backend. It prints logs to `console.error` formatting each line as: + `[Process ID] [Namespace | Severity] [Message] [Metadata JSON]` + It dynamically applies terminal colors to severity tags and metadata if color capabilities are enabled. +2. **`DebugBackend`**: A bridge adapter for applications already using the npm `debug` package, allowing unified filtering through `NODE_DEBUG`. +3. **`StructuredBackend`**: Adapts logs into standard JSON matching Google Cloud's structured logging specifications: + ```json + { + "severity": "INFO", + "message": "Formated log message", + "telemetryTraceId": "xyz...", + "other": { "extra": "metadata" } + } + ``` + Logs can be written to standard out or forwarded to an upstream backend. + +#### Registry and Lifecycle Management +- **`loggerCache`**: Tracks all active loggers by namespace. This guarantees that if multiple modules request `log('pubsub:connection')`, they receive the exact same instance, keeping event listeners intact. +- **`setBackend(backend)`**: Replaces the global backend on-the-fly and flushes the logger cache to ensure that subsequent calls utilize the new logging sink configuration. + +--- + +## πŸ§ͺ Testing Suite Details + +The library contains unit tests and sample system tests to verify the correctness and resilience of the logging layer. + +### 1. Unit Tests (`test/temporal.ts`) +Verifies the TC39-like `Duration` utility: +- Confirms `Duration.from()` handles various combinations of time values (e.g., creation from milliseconds, seconds, minutes, or hours). +- Verifies that `totalOf()` performs precise floating-point math when converting back to individual units. + +### 2. Unit Tests (`test/logging-utils.ts`) +Provides extensive functional coverage for the core logging architecture using a custom `TestSink` test mock: +- **Disabled State Verification**: Asserts that logging is completely silent when no enabling environment flags are defined. +- **Environment Parsing**: Validates that the `GOOGLE_SDK_NODE_LOGGING` variable accepts wildcards, specific systems, and handles alias variables (like `all` mapping to `*`). +- **Manual Backends**: Confirms that configuring `setBackend(sink)` manually works independently of environment flags. +- **Severity & Direct Invocation**: Tests all direct logging styles (e.g., calling `logger()`, `logger.info()`, etc.) and validates metadata values. +- **Instance Caching**: Verifies that identical namespaces resolve to the exact same object instance (`logger === logger2`). +- **Event Tapping**: Validates the `EventEmitter` functionality by checking that custom listeners receive `'log'` event envelopes correctly. +- **Structured Logs**: Tests the structured JSON output format, stubbing standard console methods with `sinon` to assert matching ingestion keys. +- **Hierarchy**: Tests that `logger.sublog('subsys')` prefixes namespaces correctly and routes records through the system hierarchy. + +--- + +## πŸš€ Sample and System-Test Integration + +To see `google-logging-utils` in action, we can review the packaged sample applications under the `samples/` directory. + +### Quickstart Script (`samples/typescript/quickstart.ts`) +Illustrates how libraries incorporate the utility: +```typescript +import {log} from 'google-logging-utils'; + +function main() { + // Obtain a namespaced logger + const test = log('testing'); + + // Structured logging with metadata + test({other: {foo: 'bar'}}, 'boo'); + + // Severity logging + test.info('info'); +} + +main(); +``` + +### System Test Integration (`samples/system-test/test.quickstart.ts`) +This system test ensures that logs behave appropriately under real-world executions by spawning separate Node processes via `child_process.execSync`: +1. **Quiet Execution Test**: + Executes the quickstart sample in an environment *lacking* the `GOOGLE_SDK_NODE_LOGGING` variable. Asserts that the output is completely clean and that no log statements are printed to stdout/stderr. +2. **Verbose Execution Test**: + Executes the quickstart sample with the environment variable `GOOGLE_SDK_NODE_LOGGING=all`. Asserts that colorized debug logs containing severity tags and structured metadata (`foo: 'bar'`) are successfully written to terminal outputs. diff --git a/core/packages/nodejs-googleapis-common/CODEMETA.md b/core/packages/nodejs-googleapis-common/CODEMETA.md new file mode 100644 index 000000000000..342b53aa9f5d --- /dev/null +++ b/core/packages/nodejs-googleapis-common/CODEMETA.md @@ -0,0 +1,254 @@ +# GoogleAPIs Common (`googleapis-common`) Package Architecture Reference +# *WARNING*: This file is AI generated and may contain inaccuracies. + +The `googleapis-common` package (`@google-cloud/googleapis-common`) is a core internal shared tooling library utilized extensively by `googleapis` and other Google API client libraries. + +> [!NOTE] +> As stated in its description, this is a helper package and is not intended for direct end-user/consumer consumption. It acts as the engine room for generated clients, providing unified configuration, dynamic API generation, request orchestration, authentication hooks, Trusted Partner Cloud (TPC) universe domain routing, and an experimental HTTP/2 connection-pooling transport layer. + +--- + +## 1. Core Architecture & Design Patterns + +The package operates as a bridge between standard HTTP client utilities (like `gaxios`), authentication frameworks (`google-auth-library`), and API definitions (Google Discovery Documents). Below is the system-level architecture: + +```mermaid +graph TD + subgraph Consumer Layer + Client["Client Application"] + end + + subgraph API Instance Binding + getAPI["getAPI() (apiIndex.ts)"] + Endpoint["Endpoint (endpoint.ts)"] + Discovery["Discovery (discovery.ts)"] + end + + subgraph Request Orchestration + createAPI["createAPIRequest() (apirequest.ts)"] + end + + subgraph Transport & Auth + auth["AuthPlus (authplus.ts)"] + http2["http2.ts (connection pooled)"] + gaxios["Gaxios (HTTP 1.1)"] + end + + subgraph Network + GoogleAPI["Google API Endpoint"] + end + + Client -- "Instantiates Client" --> getAPI + Discovery -- "Fetches Discovery Doc & generates" --> Endpoint + getAPI -- "Instantiates generated subclass of" --> Endpoint + Client -- "Invokes API method" --> Endpoint + Endpoint -- "Funnels details to" --> createAPI + createAPI -- "Checks TPC & resolves credentials via" --> auth + createAPI -- "Dispatches H2" --> http2 + createAPI -- "Dispatches H1.1" --> gaxios + http2 -- "HTTP/2 Session" --> GoogleAPI + gaxios -- "HTTP/1.1 Request" --> GoogleAPI +``` + +### Main Architectural Pillar Patterns: +1. **Dynamic API Discovery & Client Binding**: By loading or fetching JSON Google Discovery Documents, the `Discovery` and `Endpoint` classes dynamically construct nested resource namespaces and construct executable JavaScript methods on the fly. +2. **Universal Request Orchestrator**: The core `createAPIRequest` function handles parameter mapping (e.g., aliased keyword parameters), required argument validations, URL-template expansions, custom query serialization, multipart/related file upload stream-tracking, and credentials mapping. +3. **Trusted Partner Cloud (TPC) Universe Domains**: Dynamically intercepts target URLs to map Google API requests to customized partner cloud environments based on environment variables or custom library options, enforcing validation policies to prevent credential leakage between separate universes. +4. **Experimental HTTP/2 Connection Pooling**: Implements a custom lightweight HTTP/2 client featuring automated host connection pooling, proactive inactive session teardown timeouts, and stream-based data buffering. + +--- + +## 2. Package Directory Tour & Execution Flow + +When a client invokes a method on an instantiated Google API service, the execution flows through the following components: + +``` +[Client Call] + β”‚ + β–Ό +[Endpoint.makeMethod()] ──► Compiles parameter lists, defines method URLs, and checks upload capabilities. + β”‚ + β–Ό +[createAPIRequest()] ──► Merges options, checks required params, resolves TPC domains, converts headers, + β”‚ and formats browser vs. Node stream/multipart payloads. + β–Ό + (http2?) + β”œβ”€β”€β–Ί [Yes] ──────────► Resolves auth headers ──► [http2.request()] ──► Reuses pooled h2 connection. + └──► [No] ──────────► [authClient.request() or new Gaxios().request()]. +``` + +--- + +## 3. Source Code Files (`src/`) + +### [index.ts](index.ts) +- **Purpose**: The primary library entrypoint that aggregates and re-exports all public interfaces, classes, helper utilities, and crucial external dependency types (from `google-auth-library` and `gaxios`). + +### [api.ts](api.ts) +- **Purpose**: Hosts standard TypeScript interfaces and type declarations describing request configurations, option configurations, and service metadata. +- **Key Elements**: + - `APIRequestParams`: Holds parameter bags, path parameter lists, required parameter checks, context references, and optional media links. + - `GlobalOptions` & `MethodOptions`: Extends standard `gaxios` options to specify API versions, target URLs, authentication credentials, and universe domain overrides. + - `UserAgentDirective`: Models the product name, version, and comments needed to format custom `User-Agent` header lines. + - `BodyResponseCallback`: Signature for classic Node-style callback invocations. + +### [apiIndex.ts](apiIndex.ts) +- **Purpose**: Provides the `getAPI` engine instantiation factory function. +- **How it works**: Client modules pass the target API name, options/version, version registry mapping, and global context. `getAPI` resolves the designated version class constructor, instantiates the class with specified options, freezes the resulting object via `Object.freeze` to prevent runtime mutations, and returns it. + +### [apirequest.ts](apirequest.ts) +- **Purpose**: The central operational heart of the package, responsible for processing and dispatching API requests. +- **How it works**: + - Deep-merges local request parameters, API service-level options, and global Google configurations. + - Normalizes parameters; specifically, fields clashing with JS/TS reserved keywords (which generated clients suffix with an underscore like `resource_`) have their trailing underscore stripped. + - Resolves the request body, supporting legacy `resource` parameters and modern `requestBody` conventions. + - Performs required parameter validation and expands path/URL templates using `url-template`. + - Custom-serializes arrays in the querystring using `qs` in a repeating fashion (`a=1&a=2`) and replaces `+` characters with `%20` encoding. + - Handles TPC Universe domain routing, replacing `.googleapis.com` host suffixes if custom domains are resolved, and cross-validates credential metadata against the target universe to prevent token leaks. + - Processes media/multipart uploads; for Node.js, it builds stream-based `multipart/related` payloads tracked via `ProgressStream` transforms to notify consumers of upload progress. For browsers, it formats and submits memory-bound string buffers. + - Adds mandatory instrumentation headers (`x-goog-api-client`, `x-goog-api-version`). + - Dispatches the request using either experimental `http2.request`, authenticated `authClient.request`, or standalone `Gaxios.request`. + +### [authplus.ts](authplus.ts) +- **Purpose**: Wraps `GoogleAuth` from `google-auth-library` to provide a unified authentication constructor class (`AuthPlus`). +- **How it works**: Exposes core auth client classes (like `JWT`, `Compute`, `OAuth2Client`, etc.) directly on the class instance. It overrides `getClient()` and `getProjectId()` to keep a cached memo of the most recently generated authentication instance, ensuring that subsequent calls resolve project IDs within the same credential context. + +### [discovery.ts](discovery.ts) +- **Purpose**: Manages runtime API client generation from Google Discovery Documents. +- **How it works**: + - Fetches Discovery Document schemas dynamically from remote discovery URL endpoints or reads them from the local filesystem if no protocol prefix exists. + - Leverages `makeEndpoint` to produce dynamic `Endpoint` factory instantiators and uses `applySchema` to map the schema's API layout dynamically into executable namespaces in memory. + - Offers `discoverAllAPIs` to fetch a root registry list and construct a frozen version-keyed client dictionary. + +### [endpoint.ts](endpoint.ts) +- **Purpose**: The base class for dynamic API namespaces and endpoints. +- **How it works**: + - Provides `applySchema()`, which recursively navigates a discovery document's nested resources and method trees, binding them to the target class instance. + - Binds resource methods using `makeMethod()`. When executed, `makeMethod` compiles parameter collections, identifies URL path variables, identifies media upload options, and delegates execution to `createAPIRequest()`. + +### [http2.ts](http2.ts) +- **Purpose**: An experimental HTTP/2 network client layer designed to support high-throughput, low-latency multiplexed API requests. +- **How it works**: + - Manages a connection registry (`sessions`) keyed by hostname to pool active `ClientHttp2Session` connections. + - Auto-cleans idle connections using a `500ms` timeout delay following request completion, releasing file descriptors to let Node.js processes terminate gracefully. + - Maps request configurations to HTTP/2 pseudo-headers (`:path`, `:method`), writes stream, string, or JSON payloads, handles automatic GZIP gunzipping, and validates response status codes. + - Automatically evicts pooled sessions from cache if they trigger `'error'` or `'goaway'` events. + +### [isbrowser.ts](isbrowser.ts) +- **Purpose**: Simple helper returning a boolean reflecting whether the script runs inside a web browser context (`typeof window !== 'undefined'`) vs. a Node.js process. + +### [schema.ts](schema.ts) +- **Purpose**: Contains exhaustive type definitions and interfaces representing Google APIs Discovery Service schemas (e.g., `Schemas`, `SchemaItem`, `SchemaParameter`, `SchemaMethod`). + +### [util.ts](util.ts) +- **Purpose**: General utility and translation layer for headers and responses. +- **Key Elements**: + - `headersToClassicHeaders`: Converts browser standard `Headers` objects or `Array` key-value lists into plain `Record` key-value dictionaries. + - `marshallGaxiosResponse`: Reformats response metadata from `Gaxios` to expose a standard, writable, and classic HTTP/2-friendly header object representation. + +--- + +## 4. Unit Test Files (`test/`) + +The unit testing strategy verifies component isolation, option-merging policies, mock discovery document binding, and transport flows using `nock` (to mock network transactions) and `proxyquire` (to mock module dependencies). + +### [test.api.ts](test/test.api.ts) +- **What is tested**: Validates basic exports and structural validation of interfaces. +- **Strategy**: Basic import checking. + +### [test.apiIndex.ts](test/test.apiIndex.ts) +- **What is tested**: Dynamic engine instantiation inside `getAPI`. +- **Strategy**: + - Asserts that providing just a version string successfully instantiates the endpoint class. + - Asserts that version parameter keys are successfully deleted from configuration parameter objects before construction. + - Asserts that invalid argument inputs throw runtime exceptions, and constructors triggering internal exceptions surface proper error envelopes. + +### [test.apirequest.ts](test/test.apirequest.ts) +- **What is tested**: Thorough, exhaustive unit coverage of the `createAPIRequest` orchestration engine. +- **Strategy & Mocking**: + - Enforces network isolation using `nock.disableNetConnect()`. + - Implements `FakeReadable` (simulating custom chunks) and `FakeWritable` (verifying multipart boundaries and tracking upload progress) to validate Node.js stream progress reporting. +- **Key Assertions**: + - Verifies parameter resolution; specifically, that non-required `resource` payloads are treated as bodies, while required string resources successfully parse into URL query options. + - Validates `userAgentDirectives` formatting at local, service, and global configuration tiers. + - Validates URL root rewriting logic. + - Asserts that `Accept-Encoding: gzip` and instrumentation client headers (`x-goog-api-client`, `x-goog-api-version`) populate correctly. + - Validates the Trusted Partner Cloud (TPC) Universe domain translation workflow: translating hosts to custom domains, preventing leaking credentials across separate universes, and prioritizing manual parameters over environment variables. + +### [test.authplus.ts](test/test.authplus.ts) +- **What is tested**: Class bindings and client retrieval in `AuthPlus`. +- **Strategy**: Leverages `proxyquire` to intercept `'google-auth-library'` and supply stubbed auth client returns to verify correct property exposure and credential instantiation. + +### [test.discovery.ts](test/test.discovery.ts) +- **What is tested**: API client construction from discovery documents. +- **Strategy**: Uses `nock` to mock remote schema document requests, loading local JSON fixtures (like `compute-v1.json`) to assert that the generated client parses resource paths and binds dynamic fields (such as `zones`) on newly initialized endpoints. + +### [test.endpoint.ts](test/test.endpoint.ts) +- **What is tested**: Constructor option localizing and dynamic schema binding on base `Endpoint` instances. +- **Strategy**: Feeds fixture schemas to `applySchema` and asserts that endpoint methods (like `list`) compile into executable JS functions on target objects. + +### [test.http2.ts](test/test.http2.ts) +- **What is tested**: The experimental HTTP/2 connection pooling and network stack. +- **Strategy**: + - Employs `proxyquire` to intercept the native `'http2'` module and stub the `connect` handler. + - Returns a `FakeClient` subclassing `EventEmitter` that yields a customizable request duplex stream (`requestStream`). +- **Key Assertions**: + - Asserts HTTP/2 pseudo-header mapping and default GZIP decoding. + - Asserts that host connections are cached in the pooled registry and automatically cleared after the designated idle timeout. + - Verifies session eviction policies upon triggering stream errors or receiving server `goaway` packets. + - Asserts correct transmission of string, stream, and object bodies. + +### [test.isbrowser.ts](test/test.isbrowser.ts) +- **What is tested**: Validates that `isBrowser` yields `false` in standard Node.js test environments. + +### [test.schema.ts](test/test.schema.ts) +- **What is tested**: Import check verifying types schema exports. + +### [test.util.ts](test/test.util.ts) +- **What is tested**: Helper methods in `util.ts`. +- **Strategy**: Asserts correct plain-object translation of `Headers` class instances, and validates that Gaxios response mapping yields writable classic headers while preserving status codes. + +--- + +## 5. Browser Tests (`browser-test/`) + +### [test.isbrowser.ts](browser-test/test.isbrowser.ts) +- **Purpose**: Verifies that when compiled and run inside a real browser context (using Karma/Chrome/Firefox via the Webpack configuration), `isBrowser()` correctly evaluates to `true`. + +--- + +## 6. Samples Tests (`samples-test/`) + +### [samples.test.ts](samples-test/samples.test.ts) +- **Purpose**: Placeholder warning test. Since this is a core internal developer helper package, it does not provide public consumption samples; hence, this prints a warning informing the runner that no sample tests are configured. + +--- + +## 7. System Tests (`system-test/`) + +System tests assert correct live integration behavior against real Google Cloud Platform services in the cloud, demanding actual Google Cloud credentials to execute successfully. + +### [test.http2.ts](system-test/test.http2.ts) +- **What is tested**: Asserts that experimental HTTP/2 requests function reliably against actual live production Google services. +- **Strategy**: + - Uses authentic GCP service credentials. + - Performs real Cloud Storage bucket list requests (`https://storage.googleapis.com/storage/v1/b`) and asserts data resolution. + - Performs translation calls using Google Translation API POST requests. + - Performs multi-part stream uploads of string chunks to Google Cloud Storage, subsequently cleaning up by invoking live HTTP/2 DELETE commands on the created resources. + +### [test.kitchen.ts](system-test/test.kitchen.ts) +- **What is tested**: Module packaging, installation compatibility, and bundling integrity. +- **Strategy**: + - Leverages standard GCP tooling package testing (often known as "kitchen sink" tests). + - Generates a package release archive tarball using `npm pack`. + - Moves the tarball to a temporary directory, copying a clean mock consumer test app (`system-test/fixtures/kitchen`) along with it. + - Executes `npm install` in the temp path to assert dependency resolution succeeds without error. + - Compiles the application using Webpack to assert browser compilation compatibility and validates that the resulting minified browser bundle size is well within parameters (under 400KB). + +--- + +## 8. Benchmarks (`benchmark/`) + +### [bench.ts](benchmark/bench.ts) +- **Purpose**: Compares request completion speed between standard HTTP/1.1 and experimental multiplexed HTTP/2. +- **How it works**: Stubbing auth header requirements, it fires 50 concurrent requests against Google's Discovery API endpoint over both protocols, printing out individual latency records and computing the average transaction duration for each layer to evaluate H2 efficiency gains. diff --git a/core/packages/retry-request/CODEMETA.md b/core/packages/retry-request/CODEMETA.md new file mode 100644 index 000000000000..02fb736a5d38 --- /dev/null +++ b/core/packages/retry-request/CODEMETA.md @@ -0,0 +1,200 @@ +# πŸ› οΈ retry-request Architectural Reference +# *WARNING*: This file is AI generated and may contain inaccuracies. + +This document serves as a comprehensive technical reference and architectural walkthrough for the `@google-cloud/retry-request` package. It details the library's core purpose, internal execution paths, design decisions, individual files, and test coverage. + +> [!IMPORTANT] +> **Deprecation Warning:** As of July 2024, this repository and package are deprecated. Relevant retry functionalities have been consolidated into [gaxios](https://github.com/googleapis/gaxios). + +--- + +## πŸ“– Overview & Purpose + +The `@google-cloud/retry-request` package provides a lightweight, flexible HTTP request wrapper that automatically retries transient failures using exponential backoff and randomized jitter. + +Key features include: +- **Decoupled Design:** It does not ship with a built-in HTTP client. Instead, it delegates request execution to an injected HTTP request function (e.g., `teeny-request` or `request`). +- **Dual-Mode API:** Operates seamlessly in both **Callback** and **Readable Stream** modes, automatically aligning with the interface of standard HTTP client streams. +- **Smart Retry Policies:** Distinguishes between transport/network errors (no response received) and HTTP status code failures (e.g., `429 Too Many Requests`, `5xx Server Errors`). +- **Capped Exponential Backoff with Jitter:** Automatically introduces randomized jitter to avoid thundering herd problems and ensures retries respect both maximum delay boundaries and total call deadlines. + +--- + +## πŸ”„ Architectural Workflow + +The following diagram details how a request traverses `retry-request`, highlighting the differences between callback and stream modes, error handling, and backoff calculation. + +```mermaid +flowchart TD + Start([Caller invokes retryRequest]) --> Mode{Is callback passed?} + + Mode -- No --> StreamMode[Stream Mode: Return PassThrough Stream] + Mode -- Yes --> CallbackMode[Callback Mode: Return abort helper] + + StreamMode --> MakeReq[makeRequest] + CallbackMode --> MakeReq + + MakeReq --> Request[Invoke opts.request] + + Request --> ResponseReceived{Response or Network Error?} + + ResponseReceived -- Network/Socket Error --> CheckNetRetry{noResponseRetries exhausted?} + CheckNetRetry -- No --> CalcDelayNet[Calculate nextRetryDelay] + CalcDelayNet --> DelayNet[setTimeout with jitter] --> MakeReq + CheckNetRetry -- Yes --> ReturnNetError[Emit error / invoke callback] + + ResponseReceived -- HTTP Response Received --> CheckHTTPRetry{retries exhausted OR shouldRetryFn returns false?} + CheckHTTPRetry -- No --> CalcDelayHTTP[Calculate nextRetryDelay] + CalcDelayHTTP --> DelayHTTP[setTimeout with jitter] --> MakeReq + CheckHTTPRetry -- Yes --> ReturnHTTP[Emit response & pipe streams / invoke callback] +``` + +--- + +## πŸ—ΊοΈ Codebase Tour & Core Mechanics + +`retry-request` is a compact package comprising only a single source file, a TypeScript type declaration file, and a comprehensive unit test suite. + +### 1. Execution Modes +The library checks the signature of the call to determine the execution mode: +* **Callback Mode:** If a function is passed as the last argument, the library wraps the call. It returns an object exposing an `abort()` function that forwards calls directly to the underlying active request. +* **Stream Mode:** If no callback is provided, it instantiates and returns a Node.js `PassThrough` stream. Because HTTP request streams start writing payload data as soon as they connect, `retry-request` pipes the internal request stream to a temporary `delayStream`. If a retry is triggered, this stream is discarded and a new request is initiated. If the request succeeds or retries are exhausted, the buffered data in the `delayStream` is piped forward to the main `retryStream`. + +### 2. Error Categories +The library separates failures into two categories: +* **Network Failures (No HTTP Response):** DNS errors, network dropouts, or connection socket hangups. Regulated by `opts.noResponseRetries` (default is `2`). +* **HTTP Status Failures:** Successful connections that returned an HTTP response. The `opts.shouldRetryFn` determines whether to retry (defaults to retrying `1xx`, `429`, and `5xx` codes). Regulated by `opts.retries` (default is `2`). + +### 3. Exponential Backoff & Jitter Calculation +Exponential backoff prevents slamming servers with repeated retries. The delay for retry number $N$ is computed as: + +$$\text{Delay} = (\text{retryDelayMultiplier}^{N} \times 1000) + \text{jitter}$$ + +* **Jitter:** A random integer between `0` and `999` milliseconds. +* **Max Delay Cap:** The calculated delay is capped at `opts.maxRetryDelay` (default is `64` seconds). +* **Deadline Cap:** To prevent calls from stalling forever, the delay is shortened or cancelled if it would cause the total request lifecycle to exceed `opts.totalTimeout` (default is `600` seconds). + +--- + +## πŸ—‚οΈ File-by-File Technical Breakdown + +The package contains the following files: + +| File Name | Purpose | Key Implementations / Structures | +| :--- | :--- | :--- | +| [index.js](index.js) | Primary Source Code | `retryRequest()`, `getNextRetryDelay()`, `DEFAULTS` | +| [index.d.ts](index.d.ts) | TypeScript Declaration File | Module type signatures and interfaces | +| [package.json](package.json) | Dependency & Scripts Definition | Versioning, runtime scripts, dependencies | +| [test.js](test.js) | Unit Test Suite | Stream, callback, overriding, and backoff tests | + +--- + +### 1. [index.js](index.js) + +Contains the core algorithm and orchestrates the request/retry flows. + +#### Global State & Configurations +* **`DEFAULTS`:** An object defining fallback configurations: + ```js + { + objectMode: false, + retries: 2, + maxRetryDelay: 64, // Capped maximum retry delay (seconds) + retryDelayMultiplier: 2, // Multiplier base for exponential backoff + totalTimeout: 600, // Hard limit for the overall operation duration (seconds) + noResponseRetries: 2, // Retries for transport errors + currentRetryAttempt: 0, + shouldRetryFn: (response) => { ... } // Returns true for 1xx, 429, and 5xx status codes + } + ``` + +#### Primary Functions +* **`retryRequest(requestOpts, opts, callback)`** + * Determines callback vs. stream mode based on arguments. + * Normalizes options, extending user configurations over `DEFAULTS`. + * Validates that `opts.request` is provided (throwing an error if missing). + * Orchestrates the request cycles using several inner helper functions: + * `resetStreams()`: Cleans up request streams and aborts active request instances to prevent memory/socket leaks. + * `makeRequest()`: Creates the request. In stream mode, it sets up a temporary `PassThrough` (`delayStream`) and registers event listeners (`error`, `response`, `complete`, `finish`) on the underlying active stream. In callback mode, it executes `opts.request(requestOpts, onResponse)`. + * `retryAfterDelay(currentRetryAttempt)`: Calculates the next sleep interval and schedules `makeRequest` via `setTimeout`. + * `onResponse(err, response, body)`: The core evaluation function. Analyzes whether the call failed or succeeded, checks retry thresholds, and either schedules a retry or propagates the result to the user. + +* **`getNextRetryDelay(config)`** + * The standalone backoff calculation helper. Takes `maxRetryDelay`, `retryDelayMultiplier`, `retryNumber`, `timeOfFirstRequest`, and `totalTimeout` parameters. + * Uses `Math.pow(retryDelayMultiplier, retryNumber)` multiplied by `1000` plus a random jitter up to `1000ms`. + * Computes the absolute maximum allowed delay based on the deadline `totalTimeoutMs - (Date.now() - timeOfFirstRequest)`. + * Caps the result to the lowest value among the calculated exponential delay, the deadline remaining time, and `maxRetryDelayMs`. + +--- + +### 2. [index.d.ts](index.d.ts) + +Provides accurate TypeScript typings for users of the library. + +* **Features:** + * Imports types from `request` and `teeny-request`. + * Exposes function overloads representing the optional `opts` and `callback` signatures. + * Defines the interface `retryRequest.Options`, detailing all configurable properties like `shouldRetryFn`, `retries`, `maxRetryDelay`, and customizable request functions. + +--- + +### 3. [package.json](package.json) + +Defines dependencies, packaging outputs, engine requirements, and scripts. + +* **Key Details:** + * **Runtime Engines:** Requires Node.js version `>=18`. + * **Dependencies:** Relies strictly on `extend` (for deep-copy options extending) and `teeny-request` (as the modern default request executor). + * **Package Files:** Restricts publication strictly to `index.js`, `index.d.ts`, and `license` to keep the footprint minimal. + * **No System Tests:** The package contains no integration or system tests. The `system-test` script runs `"echo no system test"`. + +--- + +## πŸ§ͺ Unit Tests Breakdown: [test.js](test.js) + +The unit test suite uses `mocha` as its test framework, `assert` for assertions, and `teeny-request` to simulate request flows. + +The tests are split into five logically isolated groups: + +### Group 1: Streams (`describe('streams')`) +Validates streaming request behaviors: +* **`works with defaults in a stream`**: Verifies that making a default streaming request to a `404` URL succeeds, emits a `'response'` event, and completes cleanly. +* **`allows object mode`**: Ensures that setting `objectMode: true` successfully initializes the underlying `PassThrough` stream's internal state with object support. +* **`emits an error`**: Asserts that socket errors (e.g., DNS lookup failure on non-existent host) bubble up as `'error'` events on the stream. +* **`emits a 'request' event on each request`**: Mocks an HTTP request stream to assert that a `'request'` event is fired on every retry attempt, allowing users to monitor retries. +* **`exposes an 'abort' function to match request`**: Verifies that the returned stream implements a functional `abort()` method. +* **`works on the last attempt`**: Simulates a sequence where the first two attempts return a `503` but the third succeeds (`200`). Verifies that the internal aborts occurred on the failed request streams, and the final response triggers `'complete'`. +* **`never succeeds`**: Simulates a stream that returns `503` indefinitely. Asserts that when all retries are exhausted, the `'response'` event is emitted representing the final failed status. +* **`forwards a request error`**: Ensures that a post-response error emitted on the request stream is forwarded through to the main `retryStream`'s `'error'` listener. + +### Group 2: Callbacks (`describe('callbacks')`) +Validates request behaviors when a callback function is supplied: +* **`works with defaults with a callback`**: Asserts standard out-of-the-box functionality for callback execution on a standard HTTP request. +* **`exposes an 'abort' function`**: Verifies that in callback mode, `retryRequest` returns an object containing an `abort()` method, which delegates abortion down to the active request instance. +* **`returns an error`**: Verifies that network-level connection drops are passed as the first argument (`err`) to the user callback. + +### Group 3: Configuration Overrides (`describe('overriding')`) +Ensures all parameters can be customized or safely defaulted: +* **`should ignore undefined options`**: Verifies that passing `undefined` values inside the options object does not overwrite properties; standard defaults are utilized instead. +* **`should allow overriding retries`**: Sets `retries: 0` and asserts that no second attempts are made. +* **`should use default noResponseRetries`**: Verifies that default connection/socket retries operate when no override is supplied. +* **`should allow overriding noResponseRetries`**: Overrides `noResponseRetries: 0` and verifies that connection failures are immediately propagated without retrying. +* **`should allow overriding currentRetryAttempt`**: Sets `currentRetryAttempt: 1` and asserts that the retry offset decreases the remaining retry allowance. +* **`should allow overriding shouldRetryFn`**: Customizes the retry decision function to confirm it overrides the default HTTP status code checker. +* **`should allow overriding request`**: Validates that a completely custom request function can be injected. + +### Group 4: Default Retry Rules (`describe('shouldRetryFn')`) +Validates the default HTTP status code retry logic: +* **`should retry a 1xx code`**: Verifies that all informational status codes (`100` to `199`) trigger retries. +* **`should not retry a 2xx code`**: Verifies that all success codes (`200` to `299`) bypass retries. +* **`should not retry a 3xx code`**: Verifies that redirect codes (`300` to `399`) bypass retries. +* **`should not retry a 4xx code`**: Verifies that most client error codes (`400` to `499`, excluding `429`) bypass retries. +* **`should retry a 429 code`**: Asserts that status `429` (`Too Many Requests`) triggers a retry. +* **`should retry a 5xx code`**: Asserts that all server errors (`500` to `599`) trigger retries. + +### Group 5: Time Delay Mechanics (`describe('getNextRetryDelay')`) +Verifies the delay calculation mathematics: +* **`should return exponential retry delay`**: Asserts that delays are calculated exponentially and stay strictly within computed boundary bounds ($2^N$ seconds plus $0-1000$ms jitter). +* **`should allow overriding the multiplier`**: Asserts that a customized `retryDelayMultiplier` is correctly integrated into the calculation. +* **`should honor total timeout setting`**: Asserts that if a request has been running and the overall deadline (`totalTimeout`) is approaching, the calculation limits the delay time to fit inside the remaining window rather than throwing an out-of-bounds delay. +* **`should return maxRetryDelay if calculated retry would be too high`**: Asserts that delays are capped by `maxRetryDelay` (both default and custom overridden values) to prevent massive delay times. diff --git a/core/packages/teeny-request/CODEMETA.md b/core/packages/teeny-request/CODEMETA.md new file mode 100644 index 000000000000..700bee3f1ad1 --- /dev/null +++ b/core/packages/teeny-request/CODEMETA.md @@ -0,0 +1,158 @@ +# 🌌 Teeny-Request Package Architecture +# *WARNING*: This file is AI generated and may contain inaccuracies. + +`teeny-request` is a lightweight, modern wrapper built around `node-fetch` that mimics the API surface of the legacy and deprecated `request` library. Because many Google Cloud client libraries historically relied on `request`'s configuration schema, `teeny-request` was created as a drop-in alternative to minimize package weight while preserving backward-compatible streaming, callback patterns, proxy configurations, and connection pooling. + +--- + +## πŸ—οΈ High-Level Conceptual Flow + +The execution path of a request flows through options mapping, proxy/connection pooling agent resolution, stats telemetry, and stream/callback dispatch: + +```mermaid +flowchart TD + A["Client Call to teenyRequest(options, callback?)"] --> B{"Callback Provided?"} + + subgraph Prep ["1. Options Preparation & Mapping"] + B -- Yes --> C["Callback Mode"] + B -- No --> D["Stream Mode (Returns requestStream PassThrough)"] + C & D --> E["requestToFetchOptions(reqOpts)"] + E --> F["Normalize Headers, Body, and Querystring (qs)"] + F --> G["Resolve Agent via getAgent(uri, reqOpts)"] + end + + subgraph Stats ["2. Telemetry & Stats Tracking"] + G --> H["teenyRequest.stats.requestStarting()"] + H --> I["Increment concurrentRequests counter"] + I --> J{"Exceeds warning threshold?"} + J -- Yes --> K["Emit TeenyStatisticsWarning via process.emitWarning"] + J -- No --> L["Proceed"] + end + + subgraph Fetch ["3. HTTP Fetch Execution"] + L --> M{"Multipart Payload?"} + M -- Yes --> N["Create boundary + createMultipartStream()"] + M -- No --> O["Standard Fetch Call"] + N & O --> P["Call node-fetch with RequestInit"] + P --> Q{"Fetch Response Settled?"} + end + + subgraph Cleanup ["4. Response Conversion & Dispatch"] + Q -- Success --> R["teenyRequest.stats.requestFinished()"] + Q -- Failure --> S["teenyRequest.stats.requestFinished()"] + R --> T["fetchToRequestResponse(opts, res)"] + T --> U{"Callback Mode?"} + U -- Yes --> V["Parse Body (JSON / Text) -> Invoke Callback"] + U -- No --> W["Emit 'response' event & Pipe body stream to requestStream"] + S --> X["Callback(error) OR Emit 'error' on stream"] + end +``` + +--- + +## πŸ“ Directory Structure & Component Summary + +``` +teeny-request/ +β”œβ”€β”€ src/ +β”‚ β”œβ”€β”€ index.ts # Main entrypoint, Request options normalizer, & core fetching logic +β”‚ β”œβ”€β”€ agents.ts # Connection pool caching, Proxy agent resolution, & NO_PROXY filters +β”‚ └── TeenyStatistics.ts # Concurrency tracker & threshold violation warnings emitter +└── test/ + β”œβ”€β”€ index.ts # End-to-end request mocks, callback/stream, and body type unit tests + β”œβ”€β”€ agents.ts # Proxy configurations, NO_PROXY bypass patterns, & keepAlive caching tests + └── TeenyStatistics.ts # Stats increment/decrement & emitWarning threshold tests +``` + +--- + +## πŸ› οΈ Detailed Tour of Source Code (`src/`) + +### 1. [src/index.ts](src/index.ts) +This file acts as the primary coordinator of the library. It exposes the main `teenyRequest` function, its defaults factory, and maps request configurations to the corresponding fetch parameters. + +* **`requestToFetchOptions(reqOpts: Options)`**: Converts legacy `request` options into standard `node-fetch` `RequestInit` parameters. + * Translates `reqOpts.method` (default `GET`), `reqOpts.timeout` to `timeout`, and `reqOpts.gzip` to fetch `compress`. + * Maps request bodies: automatically serializes `json` options as strings and sets `Content-Type: application/json`, or processes binary buffers and regular strings directly. + * Handles query strings: if `useQuerystring` or `qs` is passed, it uses Node's `querystring` module to append parameters safely to the URI. + * Resolves the network agent using `getAgent(uri, reqOpts)`. +* **`fetchToRequestResponse(opts: RequestInit, res: Response)`**: Normalizes the native fetch Response into a backward-compatible structure. It maps request parameters, formats headers back into an object (from fetch's `Headers` map), and sets standard response values. +* **`createMultipartStream(boundary, multipart)`**: Generates a multipart/related request stream. Used primarily for file uploads (combining metadata JSON and media streams). +* **`teenyRequest(reqOpts, callback?)`**: Directs request execution based on usage: + * *Multipart Mode*: Handles two-part uploads asynchronously. + * *Stream Mode (no callback)*: Instantiates a `stream-events` `PassThrough` stream, deferring data piping until the user sets up a reader (`reading` event), ensuring no chunk is lost. + * *Callback Mode (with callback)*: Initiates the fetch request, handles HTTP status code `204` gracefully, automatically parses `application/json` or falls back to raw text, and triggers the callback. +* **`teenyRequest.defaults(defaults)`**: Returns a pre-configured wrapper around `teenyRequest` to apply common configurations (e.g., timeouts, headers) across multiple calls. +* **`teenyRequest.stats` & `teenyRequest.resetStats()`**: Exposes the package's telemetry tracking instance. + +### 2. [src/agents.ts](src/agents.ts) +This file is responsible for optimizing connections, pooling sockets, and managing network proxies. + +* **`pool`**: A `Map` that caches created HTTP/HTTPS agents to keep connections alive and avoid socket overhead. +* **`shouldUseProxyForURI(uri)`**: Resolves environmental proxy settings. It reads `NO_PROXY`/`no_proxy` environment variables, supports comma-separated exclusion domains and wildcard domains (`*.example.com`, `.example.com`), and filters requests accordingly. +* **`getAgent(uri, reqOpts)`**: Returns the appropriate connection agent: + * If a proxy is defined (`reqOpts.proxy` or proxy environment variables) and the destination URI is not filtered by `NO_PROXY`, it dynamically imports and returns an instance of `http-proxy-agent` or `https-proxy-agent`. + * If `reqOpts.forever` is active, it returns or initializes a cached `http.Agent` or `https.Agent` with `keepAlive: true` from the global `pool` (respecting customized socket parameters in `reqOpts.pool`). + * Otherwise, it returns `undefined`, causing `node-fetch` to rely on default system agents. + +### 3. [src/TeenyStatistics.ts](src/TeenyStatistics.ts) +This file prevents resource leaks or thread pool exhaustion from uncontrolled outbound network requests. + +* **`TeenyStatistics`**: Monitors concurrent request volumes. + * **`requestStarting()`**: Increments the `concurrentRequests` counter. If the count crosses the warning threshold, it creates a `TeenyStatisticsWarning` and logs it exactly once via `process.emitWarning()`. + * **`requestFinished()`**: Decrements the active request counter. +* **Warning Threshold Configuration Precedence**: + 1. Direct option configuration via `concurrentRequests` passed to constructor/options (setting `0` completely disables warnings). + 2. The `TEENY_REQUEST_WARN_CONCURRENT_REQUESTS` environment variable. + 3. The built-in fallback default of **5000** concurrent requests. + +--- + +## πŸ§ͺ Detailed Tour of Test Suite (`test/`) + +All tests are implemented using **Mocha** as the runner, **Sinon** for mock/stub sandboxing, and **Nock** to intercept real HTTP network connections. + +### 1. [test/index.ts](ctest/index.ts) +Contains unit tests verifying request dispatching, streams, header mapping, and body parsing configurations. + +* **JSON & Request Options Resolution**: Checks if requests retrieve JSON properly and validates option forwarding in `teenyRequest.defaults`. +* **Header Mapping & Fetch Format compatibility**: Validates that the response event object and its headers align with legacy `request` formats. Tests integration with standard browser-style `Headers` objects. +* **Error Handling**: Assures that parser errors (e.g. invalid JSON payloads) are passed directly to the caller instead of being silently dropped or over-wrapped. +* **Stream Handling & Stream Pipes**: + * Verifies that stream mode exposes a valid `PassThrough` stream. + * Validates integration with stream orchestration utilities like `pumpify` (testing `setEncoding('utf8')` and async iteration). + * Verifies that response streams are not piped to the user stream until the user requests it (`data` listener or pipe setup), preventing memory/buffer issues. +* **Agent & Proxy Injection**: Assures that the `forever` keepAlive configuration correctly creates an agent and that proxy settings are correctly resolved on the dynamic agent structure. +* **Body Parsing**: Confirms the handler supports binary `Buffer`, string payloads, and JSON inputs. +* **Telemetry Integrations**: Verifies stats tracking calls `requestStarting` and `requestFinished` under success and failure outcomes across callback, stream, and multipart request modes. + +### 2. [test/agents.ts](ctest/agents.ts) +Contains unit tests for the socket pooling, agent caching, and environment proxy filters. + +* **Default Agent**: Confirms `undefined` is returned if no special options are set. +* **Proxy Agent Generation**: Asserts that explicit request proxy values or proxy environment variables (`http_proxy`, `https_proxy`, etc.) correctly generate instances of `HttpProxyAgent` or `HttpsProxyAgent`. +* **`no_proxy` and `NO_PROXY` Filters**: Asserts that proxy exclusions bypass the proxies, returning `undefined` instead. Explicitly tests wildcard matches (`*.domain`), dotted subdomains (`.domain`), list separations (`domain-a,domain-b`), and combinations thereof. +* **Keep-Alive Caching (`forever`)**: Asserts that `forever: true` produces a pooled HTTP/HTTPS agent, and verifies that subsequent calls return the exact same agent reference (avoiding new socket allocations). +* **Connection Pool Options**: Verifies that configurations under `reqOpts.pool` (like `maxSockets`) are correctly forwarded to the agent instances instead of leaking into or modifying global agents. + +### 3. [test/TeenyStatistics.ts](ctest/TeenyStatistics.ts) +Tests the telemetry thresholds and concurrency warnings behavior. + +* **Constructor & Config Precedence**: Validates the priority order (constructor options > environment variables > default `5000` value). +* **Options Mutators**: Verifies that `getOptions()` returns copies of state to prevent unexpected external mutations, and tests `setOptions()` resets. +* **Concurrency Counters**: Confirms that the active request counter increments on `requestStarting()` and decrements on `requestFinished()`. +* **Threshold Warning Triggering**: + * Verifies that exceeding the threshold triggers a `TeenyStatisticsWarning` through `process.emitWarning`. + * Verifies that warnings are emitted exactly once, avoiding console log pollution. + * Tests "yo-yoing" scenarios where active requests drop back below the threshold and rise again, assuring that warnings are not re-emitted. + * Validates the specific warning message and metadata contents (value, threshold, type). + +--- + +## 🌐 Integration & System Testing + +As specified in [package.json](cpackage.json#L21): +```json +"system-test": "echo no system test" +``` +There is **no direct integration/system test suite** within this folder. This is a deliberate design choice. `teeny-request` is a low-level dependency wrapper. It is integration-tested downstream, directly powered by the integration suites of the heavy libraries that depend on it (such as `@google-cloud/common`, `@google-cloud/storage`, etc.). Ensuring its unit tests cover all options combinations with precise HTTP mocking (`nock`) guarantees stability. diff --git a/core/packages/tools/CODEMETA.md b/core/packages/tools/CODEMETA.md new file mode 100644 index 000000000000..83d6f7c41869 --- /dev/null +++ b/core/packages/tools/CODEMETA.md @@ -0,0 +1,200 @@ +# GAPIC Tools (`gapic-tools`) Technical Architecture Reference +# *WARNING*: This file is AI generated and may contain inaccuracies. + +Welcome to the technical architectural guide for `gapic-tools` (located at `core/packages/tools`). This package provides essential build utilities, protobuf compilation pipelines, and Babel-based transpilation plugins used across the `google-cloud-node` client libraries. + +--- + +## πŸ“– Table of Contents +1. [Overview & Purpose](#-overview--purpose) +2. [High-Level Architecture](#-high-level-architecture) +3. [The Code Tour: Source Code Walkthrough](#-the-code-tour-source-code-walkthrough) +4. [The Code Tour: Test Suite Walkthrough](#-the-code-tour-test-suite-walkthrough) +5. [How to Run and Dev](#-how-to-run-and-dev) + +--- + +## 🌟 Overview & Purpose + +The `gapic-tools` package contains a set of specialized command-line interface (CLI) tools and Babel AST plugins designed to automate the packaging and building of Google Cloud Client libraries. + +Its primary duties are: +1. **Protobuf Compilation & Optimization:** Resolving `.proto` files, compiling them into unified static Javascript modules (`.js`, `.cjs`), generating corresponding TypeScript definitions (`.d.ts`), and post-processing the outputs (such as JSDoc formatting and type loosening). +2. **Asset Packaging:** Minifying generated protobuf formats and localizing/bundling common Google APIs protobuf files (`google/api/*`, `google/iam/*`, etc.) so that the library is fully self-contained before publishing. +3. **ESM and CommonJS Dual-Targeting & Mocking Conversion:** Facilitating codebases targeting both CommonJS (CJS) and EcmaScript Modules (ESM) via custom Babel AST transformations that toggle ESM flags, replace ESM-specific path resolvers with CJS equivalents, and rewrite module mocking imports (`esmock` to `proxyquire`). + +--- + +## πŸ“ High-Level Architecture + +The toolchain is split into two logical functional components: + +```mermaid +graph TD + subgraph "Proto Toolchain & Bundler" + A[prepublish.ts] -->|Copies common protos| B[./protos/google/] + C[listProtos.ts] -->|Scans protos directory| D[src/protosList.json] + E[compileProtos.ts] -->|Parses _proto_list.json| F[List of Protos] + F -->|Invokes pbjs| G[protos.js / protos.cjs] + F -->|Invokes pbts| H[protos.d.ts] + G -->|Fixes imports & license| G + H -->|Updates enums, longs, bytes types| H + I[minify.ts] -->|Minifies JS & JSON in-place| G + end + + subgraph "Babel AST Plugins" + J[replaceESMMockingLib.ts] -->|esmock βž” proxyquire| K[Test Module Mocking] + L[replaceImportMetaUrl.ts] -->|import.meta.url βž” __dirname| M[CJS Path Resolution] + N[toggleESMFlagVariable.ts] -->|isEsm βž” false/true| O[Build-Time Constants] + end +``` + +--- + +## 🚢 The Code Tour: Source Code Walkthrough + +Here is an itemized breakdown of every source file in `src/`, detailing what it does, how it does it, and its role in the ecosystem. + +### 1. `src/compileProtos.ts` +* **Purpose:** Compiles Google API proto files listed under JSON files matching `*_proto_list.json` into static JavaScript modules (`.js` and `.cjs`), generates `.d.ts` typings, and sanitizes/normalizes imports and typings. +* **Execution CLI Name:** `compileProtos` +* **How it works:** + 1. Scans input directories recursively for `_proto_list.json` files. + 2. Parses these JSON lists to obtain exact paths to the proto files, adding common Google GAX protos path dependencies automatically. + 3. Executes the `protobufjs-cli/pbjs` tool programmatically to compile the proto files to a static module (JSON or JS formats). + 4. Modifies the generated JS/CJS files using string replacements to re-export the `protobufjs/minimal` runtime via the `google-gax` library (preventing duplicate runtime definitions and keeping dependencies minimal). + 5. Programmatically executes the `pbts` CLI to convert the generated JS file into TypeScript definitions (`.d.ts`). + 6. Modifies the output `.d.ts` definitions in-place to support highly flexible types: + - Relaxes **Enums** so that fields typed as enums can accept both enum numeric values or their string representation keys (e.g., `enumField: E | keyof typeof E`). + - Relaxes **Bytes** so that they accept `Uint8Array | Buffer | string` (allowing base64 encoded representations). + - Relaxes **Longs** (Int64) so that they accept `number | Long | string` (allowing string representation of huge numbers). + - Adapts the imports (`import Long = require("long")`) to resolve cross-module issues. + +### 2. `src/listProtos.ts` +* **Purpose:** Automatically indexes all `.proto` files inside a library's `protos` folder and writes them to a JSON index catalog. +* **Execution CLI Name:** `listProtos` +* **How it works:** + - Synchronously walks the specified `protos` directory. + - Filters and maps all files ending with `.proto`. + - Outputs the catalog as a formatted JSON array into `src/protosList.json` within the targeted package. This is typically loaded at runtime by the gRPC client to map schema locations. + +### 3. `src/minify.ts` +* **Purpose:** Minifies static protobuf output assets (`.json` and `.js` files) to reduce package size and bundle size for target runtimes. +* **Execution CLI Name:** `minifyProtoJson` +* **How it works:** + - Scans files in `build/protos` or a custom folder. + - Uses `uglify-js` to shrink the content. + - > [!NOTE] + > For `.json` files, it uses specific minify settings (`expression: true`, `compress: false`, `output.quote_keys: true`) so that it strips all indentation and newlines but leaves it a syntactically valid, parsable JSON string. + > For `.js` files, it runs typical minification. + +### 4. `src/prepublish.ts` +* **Purpose:** Downloads/bundles common Google proto dependencies into the package's local build before publishing. +* **Execution CLI Name:** `prepublishProtos` +* **How it works:** + - Deletes and recreates the `./protos/google` subfolder. + - Obtains paths to Google's common proto files (`api`, `iam/v1`, `logging/type`, `monitoring/v3`, `longrunning`, `protobuf`, `rpc`, `type`, `cloud/location`) using the package `google-proto-files`. + - Copies these directories into the target library's `./protos/google` path so that the client library does not depend on external filesystem layouts at runtime. + +### 5. `src/replaceESMMockingLib.ts` +* **Purpose:** A Babel plugin designed to convert ES modules dynamic mocking libraries to CommonJS mocking libraries during dual-module test suite transformations. +* **How it works:** + - Acts as a custom Babel visitor plugin. + - **Imports:** Rewrites imports matching the source value `fromLibName` (default `esmock`) to import from `toLibName` (default `proxyquire`). + - **Call Expressions:** Intercepts call expressions where the callee identifier is `esmock`. It swaps the callee target to `proxyquire` and updates parent AST nodes appropriately. + +### 6. `src/replaceImportMetaUrl.ts` +* **Purpose:** A Babel plugin converting ESM path/directory resolution statements to CJS counterparts. +* **How it works:** + - Searches for the specific ESM AST pattern: `path.dirname(fileURLToPath(import.meta.url))`. + - Converts the entire statement into a replacement variable AST node, which is configured to `__dirname` by default. + +### 7. `src/toggleESMFlagVariable.ts` +* **Purpose:** A Babel plugin that allows toggle/injection of compile-time boolean constants (e.g., `const isEsm = true;` βž” `const isEsm = false;`) during bundle conversions. +* **How it works:** + - Targets `VariableDeclarator` nodes. + - Matches variable names configured by the user (default: `isEsm`). + - If the initialized value is a boolean literal, it updates that boolean value dynamically (default: `false`). + +--- + +## πŸ§ͺ The Code Tour: Test Suite Walkthrough + +All files under `test/` are granular unit tests written in TypeScript, executed via **Mocha**, using **c8** for coverage analysis. + +> [!IMPORTANT] +> **Note regarding System Tests:** There are **no system-test or integration-test files** in this package. All verification is done via highly detailed unit tests that run on mock files and AST constructs. The `package.json` system-test script is intentionally configured to `echo 'no system-test'`. + +Here is the exact breakdown of each unit test suite: + +### 1. `test/compileProtos.ts` +* **Tests:** `src/compileProtos.ts` +* **What it validates:** + - **Isolation & Safety:** Copies fixture files (`test/fixtures/*`) into a temporary isolated sandbox `.compileProtos-test` for each test, runs compilation within that directory, and cleans it up afterwards to avoid environment pollution. + - **GAX Path Integration:** Verifies that the default `gaxProtos` is correctly resolving to standard Google proto structures inside node modules. + - **Standard Compilation Pipeline:** Runs compilation on test directories, asserting that the program outputs valid `protos.json`, `protos.js`, and `protos.d.ts` files. It loads the generated JSON with the real `protobufjs` parser to check for missing messages or services. + - **ESM Option Support:** Asserts that passing `--esm` generates separate `protos.cjs` (CommonJS module) and `protos.js` (ESM module with standard `import` statements) properly. + - **JSON Skipping:** Verifies that the `--skip-json` option successfully skips the creation of the `.json` schema asset while still generating `.js` and `.d.ts`. + - **Typings Modification Correctness:** Validates that all relaxed union types are formatted perfectly (e.g., Enums can receive enum keys/strings, Bytes can receive Buffer/string, Longs can receive strings/numbers). + - **Root Name Guessing:** Checks that `generateRootName()` automatically extracts the package name from nearby `package.json` files to form unique static root references, falling back to a hashed string ID if none is found. + - **JSDoc Transformation:** Verifies that legacy JSDoc link hashes (e.g., `{@link Service#Method}`) are reformatted to pipe syntax (`{@link Service|Method}`) which is standard for Google docs. + - **Keep-case and Force-number Flags:** Validates that options `--keep-case` and `--force-number` are fully respected when compiling, checking snake_case keys and numeric types respectively. + +### 2. `test/minify.ts` +* **Tests:** `src/minify.ts` +* **What it validates:** + - **JSON Minification:** Copies `echo.json` to the test sandbox, runs minification, and checks that the file size is reduced significantly while verifying that the parsed JSON object structure remains exactly identical. + - **JS Minification:** Copies `echo.js` to the test sandbox, minifies it, and ensures the exported JS functions and behaviors remain completely intact while the byte size decreases. + +### 3. `test/replaceESMMockingLib.ts` +* **Tests:** `src/replaceESMMockingLib.ts` +* **What it validates:** + - Compiles dynamic test strings of JavaScript and runs them through Babel with the custom `replaceESMMockingLib` plugin. + - Verifies default replacements of `import esmock from 'esmock'` and `const foo = await esmock()` down to CJS equivalents. + - Validates custom configurations specifying custom target names and library names, and checks that non-matching statements are ignored. + +### 4. `test/replaceImportMetaUrl.ts` +* **Tests:** `src/replaceImportMetaUrl.ts` +* **What it validates:** + - Transforms sample code using Babel and the plugin. + - Ensures that standard directory operations like `path.dirname(fileURLToPath(import.meta.url))` are swapped for `__dirname`. + - Assures that invalid/incomplete references (such as `import.meta` or custom properties like `import.meta.foo`) are ignored and left alone. + - Checks that custom replacement targets (like replacing with `foo.bar`) function correctly. + +### 5. `test/toggleESMFlagVariable.ts` +* **Tests:** `src/toggleESMFlagVariable.ts` +* **What it validates:** + - Feeds sample variables (e.g., `const isEsm = true;`) through Babel with the plugin. + - Asserts that the boolean literal value is replaced correctly (e.g., converted to `false`). + - Verifies that non-boolean variables (e.g., `const isEsm = 100;`) are not modified. + - Ensures custom identifier matches (e.g., target variable named `foo` instead of `isEsm`) are successfully toggled. + +--- + +## πŸš€ How to Run and Dev + +To compile, lint, and run unit tests for `gapic-tools`, use the following workspace commands: + +### Setup & Clean +```bash +# Clean up compiled artifacts +npm run clean + +# Build/compile the TypeScript files to Javascript (build/src/) +npm run compile +``` + +### Linting +```bash +# Lint files under src/ and test/ +npm run lint + +# Automatically fix style/formatting issues +npm run fix +``` + +### Testing +```bash +# Prepares the build and runs all mocha tests under build/test +npm test +``` diff --git a/core/paginator/CODEMETA.md b/core/paginator/CODEMETA.md new file mode 100644 index 000000000000..732e9524fd68 --- /dev/null +++ b/core/paginator/CODEMETA.md @@ -0,0 +1,154 @@ +# Architectural Reference: @google-cloud/paginator +# *WARNING*: This file is AI generated and may contain inaccuracies. + +This directory contains the `@google-cloud/paginator` package, a lightweight utility designed to simplify pagination across Google Cloud Node.js client libraries (such as `@google-cloud/storage`, `@google-cloud/pubsub`, `@google-cloud/bigquery`, etc.). + +It wraps standard cursor-based/paginated API calls to provide: +1. **Auto-Pagination**: Transparently fetches all pages of results and yields them via a single Callback or Promise. +2. **Streaming Pagination**: Exposes an active, readable Node.js object-mode `Transform` stream that fetches pages on-demand as the consumer reads, gracefully managing backpressure and API limits. + +--- + +## πŸ—οΈ Architectural Overview + +The package converts page-by-page request-response operations into continuous streams or buffered collections. + +Below is a sequence diagram illustrating how `paginator` wraps a typical paginated client method to provide auto-pagination: + +```mermaid +sequenceDiagram + autonumber + actor Client as Developer Application + participant Proto as Class Prototype (Extended) + participant Pag as Paginator (index.ts) + participant Stream as ResourceStream (resource-stream.ts) + participant Original as Original API Method + + Client->>Proto: method(query, callback) + activate Proto + Proto->>Pag: parseArguments_(args) + Note over Pag: Extracts query, autoPaginate, maxResults,
maxApiCalls, and callback + Pag-->>Proto: parsedArguments + + alt autoPaginate is false + Proto->>Original: Invoke directly + Original-->>Client: Custom page results + else autoPaginate is true + Proto->>Pag: run_(parsedArguments, originalMethod) + activate Pag + Pag->>Pag: runAsStream_(parsedArguments, originalMethod) + Pag->>Stream: New ResourceStream(parsedArguments, originalMethod) + activate Stream + Stream-->>Pag: ResourceStream Instance + deactivate Stream + + Note over Pag: Registers error, data, & end listeners
to accumulate results in memory + + loop For each page requested by Stream + Stream->>Original: originalMethod(nextQuery, cb) + activate Original + Original-->>Stream: cb(err, pageResults, nextQuery, metadata) + deactivate Original + Note over Stream: Pushes individual results into stream buffer,
decrements maxResults cap + end + + Stream-->>Pag: 'end' event emitted + Pag-->>Client: callback(null, allResults, query, responseMetadata) + deactivate Pag + end + deactivate Proto +``` + +--- + +## πŸ“ Source Files (`src/`) + +### πŸ“„ [index.ts](src/index.ts) +The entry point that defines the main `Paginator` orchestration class. + +* **Role**: Monkey-patches client class prototypes, parses method arguments, and switches between buffered auto-pagination and raw manual pagination. +* **Key Mechanisms**: + * **`extend(Class, methodNames)`**: Intercepts prototype methods. It caches the original method under `methodName_` and overrides the original method with a wrapper that parses arguments and calls `run_`. + * **`streamify(methodName)`**: Creates a stream factory method on a class prototype, wrapping calls to return a `ResourceStream` directly. + * **`parseArguments_(args)`**: Parses arguments passed to API calls to handle multiple signatures (e.g. optional queries or callback functions). It handles the following pagination caps and conventions: + * **`maxResults` / `pageSize`**: Detects custom result limits. If a limit is detected or if `autoPaginate` is explicitly set to `false`, `autoPaginate` defaults to `false` so pages are not auto-fetched in the background. + * **`maxApiCalls`**: Detects the maximum number of API calls (pages) to request. + * **`streamOptions`**: Sanitizes user query options by stripping paginator-specific flags (`autoPaginate`, `maxResults`, `pageSize`) and passing the remaining options to the stream constructor. + * **`run_(parsedArguments, originalMethod)`**: If `autoPaginate` is false, it bypasses the wrapper and executes the original method. If `autoPaginate` is true, it runs the paginator as a stream using `runAsStream_`, captures all data in memory, and: + * If a **Callback** was provided: Invokes the callback with compiled results, query parameters, and raw response metadata. + * If **no Callback** was provided: Returns a Promise that resolves with `[allResults, query, ...rawResponseMetadata]`. + +--- + +### πŸ“„ [resource-stream.ts](src/resource-stream.ts) +An object-mode Node.js `Transform` stream wrapper for cursored API methods. + +* **Role**: Manages on-demand asynchronous fetching of paginated resources, enforcing backpressure and limit caps. +* **Key Mechanisms**: + * **Constructor**: Initialized in `objectMode: true`. Stores the query state, caps (`maxApiCalls`, `resultsToSend` derived from `maxResults`), and original request function. + * **`_read()`**: + * Safe-guards against re-entrant calling by tracking `_reading` state. + * Wraps the API function invocation in a `try-catch` block so that synchronous validation or payload preparation errors (common in services like BigQuery) are caught and emitted as stream errors (`this.destroy(err)`). + * Executes the API call. In the callback, it: + * Decrements `_resultsToSend` and splices the fetched array to match user-requested limits. + * Iterates through the page results, pushing items using `this.push()`. If `push()` returns `false` (signaling downstream backpressure), iteration pauses. + * Verifies if pagination is complete (no `_nextQuery` cursor exists, or `_resultsToSend <= 0`, or the max API call count `_maxApiCalls` has been exceeded). If so, calls `this.end()`. + * If more results are available and the stream buffer is not full, schedules another page fetch using `setImmediate(() => this._read())`. + +--- + +## πŸ§ͺ Test Suites (`test/`) + +The testing suites reside in `test/` and fully cover the paginator and resource stream mechanics under Mocha. + +### πŸ“„ [test/index.ts](src/test/index.ts) +Unit tests focusing on prototype monkey-patching (`extend`), stream factory generation (`streamify`), argument parsing (`parseArguments_`), and callback/promise resolution buffering (`run_`). + +* **Testing Strategy**: Mocks `ResourceStream` with a `FakeResourceStream` via `proxyquire` to isolate prototype and array-buffering mechanics. +* **Key Test Coverage**: + * **`extend`**: + * Overwriting prototype methods and caching them as `method_`. + * Accepting both arrays of strings and single strings for method names. + * Maintaining correct execution context (`this`) when invoking original methods. + * **`streamify`**: + * Instantiating and returning readable streams. + * Sourcing the correct underlying method (specifically trying the cached private `method_` first). + * **`parseArguments_`**: + * Setting query defaults, detecting callback functions regardless of placement (first or last argument). + * Mapping Pub/Sub `pageSize` convention and standard `maxResults`/`maxApiCalls`. + * Ensuring `autoPaginate` falls back to `false` if max limits or explicit disables are supplied. + * Stripping paginator configuration details from raw `streamOptions` passed to the stream constructor. + * **`run_`**: + * Buffer-on-end logic when `autoPaginate` is true (for both callback and promise execution paths). + * Propagating error events to rejection handlers or callback errors. + * Returning additional page-response metadata (captured in `_otherArgs`) back to caller callbacks/promise resolutes. + * Directly calling underlying methods without buffering when `autoPaginate` is false. + +--- + +### πŸ“„ [test/resource-stream.ts](src/test/resource-stream.ts) +Unit tests verifying the core pagination stream lifecycle, limits, and backpressure. + +* **Testing Strategy**: Employs Sinon sandboxing, spies, and fake timers to mock async paging intervals and watch stream execution. +* **Key Test Coverage**: + * **`instantiation`**: Passing configuration options (e.g., `highWaterMark`) to the Node.js stream constructor, verifying initialization states (`_reading = false`, `_ended = false`). + * **`end`**: Setting correct state flags and verifying proper subclass delegation to `Transform.prototype.end`. + * **`_read`**: + * Ensuring concurrent reads are blocked (`_reading` lock-out). + * Handling async request callback arguments (extracting cursors, error propagation, and logging response metadata to `_otherArgs`). + * Adjusting `_resultsToSend` counter and splicing result arrays to match limits. + * Confirming that stream stops pushing if `end()` is called mid-push. + * Stopping pagination when cursors run dry, or when hitting `maxResults`/`maxApiCalls` caps. + * Verifying backpressure: when downstream buffer is full (e.g. high water mark reached), pagination halts until consumed. + * Rescheduling next reads using `setImmediate()` only when stream is active and has buffer space. + * Catching synchronous exceptions thrown from user request functions and safely routing them to the `'error'` stream event. + +--- + +## 🌐 Integration & System Tests (`system-test/`) + +> [!NOTE] +> This sub-package does **not** maintain a dedicated `system-test` directory. +> +> As configured in [package.json](package.json#L21), the `system-test` script simply outputs `no system test`. +> Because `@google-cloud/paginator` is a pure in-memory logical utility and stream orchestration wrapper (having no direct dependencies on external Google Cloud APIs or network environments), integration validation is handled directly inside the system tests of downstream libraries that consume it (e.g. `@google-cloud/storage`).