Add smokescreen HTTP CONNECT proxy support

[tonyxiao]

Summary

• Add docker/smokescreen/Dockerfile — builds smokescreen from source (stripe/smokescreen) via multi-stage Go build
• Add smokescreen service to compose.yml (opt-in via profiles: [smokescreen], port 4750)
• Patch https.globalAgent with HttpsProxyAgent at engine startup so googleapis/gaxios routes through the proxy when HTTPS_PROXY is set
• Configure Stripe SDK with Stripe.createNodeHttpClient(new HttpsProxyAgent(proxyUrl)) in makeClient() — the Stripe SDK ignores https.globalAgent and needs an explicit agent
• Add e2e/smokescreen.test.sh — starts smokescreen, sets HTTPS_PROXY=http://localhost:4750, then verifies src-stripe read + dest-pg write + dest-sheets write all work through the proxy
• Wire smokescreen.test.sh into the CI e2e job

Test plan

• pnpm build passes
• pnpm lint passes
• pnpm format:check passes
• CI shell wiring check passes
• e2e/smokescreen.test.sh passes in CI e2e job (requires STRIPE_API_KEY secret)

🤖 Generated with Claude Code

In general, to fix this kind of issue you should avoid returning raw exception details (messages, stacks, or serialized Error objects) to the client. Instead, log full details on the server for debugging, and return a generic, non-sensitive error description and optionally a stable error code or identifier.

For this specific code, the most targeted, low-impact fix is:

• Keep logging the detailed errorMessage on the server so debugging remains possible.
• Continue to store the detailed errorMessage in the database for internal diagnostics if that’s desired.
• Change only the HTTP response in the backfill error path so that it does not echo errorMessage back to the client.
• Optionally, return a generic message and/or a simple error code that reveals nothing about internals.

Concretely:

• In handleBackfill’s catch block (around lines 886–907), replace return jsonResponse({ error: errorMessage }, 500) with a generic response such as return jsonResponse({ error: 'Internal error in backfill worker' }, 500) (or even just 'Internal server error'). This preserves the function’s behavior (still returns HTTP 500 and JSON with an error field) but stops leaking potentially sensitive exception data.
• No changes are needed to jsonResponse itself; limiting what is passed into it is sufficient.

[Copilot]

Pull request overview

Adds opt-in Smokescreen (HTTP CONNECT) proxy support and routes outbound HTTPS traffic through it when HTTPS_PROXY/HTTP_PROXY is set, while also introducing substantial new infrastructure and documentation scaffolding (engine/service apps, Temporal integration, docs site build, and a schema visualizer).

Changes:

• Add Smokescreen build + compose.yml profile, and patch Node’s global HTTPS agent to use HttpsProxyAgent.
• Introduce new apps/engine and apps/service packages (including Temporal workflows/activities) plus supporting tests and build configs.
• Replace/expand the docs system (Markdoc build, many new architecture/spec docs) and add a Next.js visualizer app + demos/scripts.

Reviewed changes

Copilot reviewed 151 out of 501 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
docs/pages/tsconfig.md	New documentation describing TS nodenext configuration and tradeoffs
docs/pages/service/sync-types.ts	Adds documented type models for service-layer credential/sync config
docs/pages/service/sync-examples.ts	Adds TS examples + a minimal JSON-file-backed SyncAPI mock
docs/pages/service/sync-examples.sh	Adds shell script demonstrating SyncAPI CRUD via ts-cli
docs/pages/service/sync-api.ts	Adds route-map typing for SyncAPI endpoints
docs/pages/service/scenarios.md	Adds scenario-driven validation checklist for service layer
docs/pages/service/entities.puml	Adds UML/JSON diagrams for Sync/Credential entities
docs/pages/service/cli.md	Documents engine-layer CLI and pipe mode commands
docs/pages/service/ARCHITECTURE.md	Documents service/engine/source/destination layering
docs/pages/publishing.md	Documents packaging/publish strategies and Docker hybrid approach
docs/pages/plan-008-scope-rename-stripe-sync.md	Adds plan doc for npm scope rename
docs/pages/plan-007-supabase-fan-out-implementation.md	Adds plan doc for Supabase fan-out backfill rewrite
docs/pages/plan-005-cli-progress-display.md	Adds plan doc for CLI progress display
docs/pages/plan-002-move-openapi-to-source-stripe.md	Adds plan doc for moving OpenAPI logic into source-stripe
docs/pages/local-registries.md	Documents Verdaccio local registry usage and compose profile
docs/pages/index.md	New docs landing page (Markdoc) with quick start + links
docs/pages/engine/sync-engine-types.ts	Documents message protocol TS types for the engine
docs/pages/engine/sync-engine-api.ts	Documents engine interfaces (Source/Destination/Orchestrator)
docs/pages/engine/state-flow.puml	Adds sequence diagram for state flow/checkpointing
docs/pages/engine/scenarios.md	Adds concrete engine interface scenarios and validation checks
docs/pages/engine/protocol.md	Adds detailed protocol writeup and Airbyte compatibility notes
docs/pages/cli-design-lessons.md	Adds design notes for CLI/API design heuristics
docs/pages/cli-config-resolution.md	Adds CLI config resolution specification
docs/package.json	Adds docs package configuration for Markdoc build/dev
docs/openapi/webhook.json	Adds OpenAPI spec for webhook ingress endpoint
docs/layout.html	Adds docs HTML layout and Mermaid bootstrap script
docs/index.md	Removes old docs index (legacy markdown)
docs/edge-function.md	Removes legacy Supabase edge-function markdown doc
docs/docker.md	Removes legacy Docker deployment markdown doc
docs/d2.mjs	Adds D2 diagram rendering helper and CLI
docs/contributing.md	Removes legacy contributing doc under docs/
docs/build.mjs	Adds Markdoc-based static docs build script
docs/.gitignore	Ignores Vercel directory for docs deploy
docker/smokescreen/Dockerfile	Adds multi-stage build to compile Smokescreen from source
docker/compose.yml	Removes old minimal docker compose file under docker/
demo/write-to-sheets.sh	Adds demo script to write NDJSON records to Google Sheets connector
demo/write-to-postgres.sh	Adds demo script to write NDJSON records to Postgres connector
demo/stripe-to-postgres.sh	Adds demo script to run a small Stripe→Postgres sync
demo/stripe-to-google-sheets.sh	Adds demo script to run a small Stripe→Sheets sync
demo/reset-postgres.sh	Adds helper to drop non-system schemas/tables locally
demo/read-from-stripe.sh	Adds demo script to read NDJSON from Stripe connector
compose.yml	Adds root compose with core services + Temporal + Smokescreen + Verdaccio profiles
apps/visualizer/src/app/page.tsx	Adds Next.js visualizer page with client-only loading skeleton
apps/visualizer/src/app/layout.tsx	Adds Next.js root layout + metadata
apps/visualizer/src/app/globals.css	Adds Tailwind import
apps/visualizer/postcss.config.mjs	Adds Tailwind postcss config
apps/visualizer/package.json	Adds visualizer package dependencies (Next/React/PGlite/CodeMirror)
apps/visualizer/next.config.ts	Enables async WASM + COEP/COOP headers for browser WASM usage
apps/visualizer/README.md	Documents visualizer data generation + run workflow
apps/visualizer/.gitignore	Ignores TS build info in app
apps/supabase/vitest.integration.config.ts	Adds separate Vitest config for integration tests
apps/supabase/vitest.e2e.config.ts	Adds separate Vitest config for e2e tests + timeout
apps/supabase/vitest.config.ts	Updates default vitest exclusions to keep suites separate
apps/supabase/tsconfig.json	Adds nodenext/tsc output config for Supabase app
apps/supabase/src/lib.ts	Fixes ESM import extension (./supabase.js)
apps/supabase/src/index.ts	Adds barrel export for Supabase app modules
apps/supabase/src/edge-functions/deno.json	Changes Deno import mapping (postgres→pg)
apps/supabase/src/edge-function-code.ts	Adds ?raw import for bundled edge function code
apps/supabase/src/tests/supabase.e2e.test.ts	Adds Supabase e2e tests against real project + Stripe
apps/supabase/src/tests/install.e2e.test.ts	Adds install/uninstall e2e test (not enabled in CI yet)
apps/supabase/src/tests/edge-runtime.smoke.test.ts	Adds edge-runtime container smoke test
apps/supabase/src/tests/deploy.e2e.test.ts	Adds edge function deploy/invoke/delete e2e test
apps/supabase/src/tests/bundle.test.ts	Adds bundled edge function code quality tests
apps/supabase/package.json	Adds package metadata, build/test scripts, deps and devDeps
apps/supabase/build.mjs	Reworks build to use esbuild bundling + tsc --emitDeclarationOnly
apps/service/tsconfig.json	Adds nodenext/tsc output config for service app
apps/service/src/temporal/workflows.ts	Adds Temporal workflow orchestration for sync lifecycle
apps/service/src/temporal/worker.ts	Adds Temporal worker factory
apps/service/src/temporal/types.ts	Adds workflow/activity shared types
apps/service/src/temporal/bridge.ts	Adds bridge between service and Temporal WorkflowClient
apps/service/src/temporal/activities.ts	Adds Temporal activities calling engine API + parsing NDJSON
apps/service/src/lib/stores.ts	Adds store interfaces for config/state/credentials/logging
apps/service/src/lib/stores-fs.ts	Adds file-backed store implementations
apps/service/src/lib/schemas.ts	Adds zod schemas for credentials/sync configs/log entries
apps/service/src/lib/resolve.ts	Adds config+credential+state resolution into engine SyncParams
apps/service/src/index.ts	Adds public barrel exports for @stripe/sync-service
apps/service/src/cli/index.ts	Adds CLI entry re-export
apps/service/src/api/webhook-app.ts	Adds standalone webhook ingress app + OpenAPI docs
apps/service/src/api/index.ts	Adds API barrel export
apps/service/package.json	Adds service package metadata, exports, and deps (incl. Temporal optional deps)
apps/engine/vitest.config.ts	Adds vitest config for engine app with longer timeouts
apps/engine/tsconfig.json	Adds nodenext/tsc output config for engine app
apps/engine/src/serve-command.ts	Adds serve action wiring connector resolution + app
apps/engine/src/lib/state-store.ts	Adds engine state store interface + noop implementation
apps/engine/src/lib/source-test.ts	Adds test source connector
apps/engine/src/lib/source-exec.ts	Adds subprocess Source wrapper for connector CLIs
apps/engine/src/lib/select-state-store.ts	Adds convention-based dynamic state-store selection
apps/engine/src/lib/resolver.test.ts	Adds tests for connector resolver behavior
apps/engine/src/lib/pipeline.ts	Adds pipeline stages (catalog enforcement, logging, filtering, persistence)
apps/engine/src/lib/ndjson.ts	Adds NDJSON parsing helpers for string/chunks/streams
apps/engine/src/lib/ndjson.test.ts	Adds thorough tests for NDJSON parsing
apps/engine/src/lib/index.ts	Adds engine lib barrel exports (protocol + helpers + test connectors)
apps/engine/src/lib/exec.test.ts	Adds tests for exec-wrapped connectors + error propagation
apps/engine/src/lib/exec-helpers.ts	Adds spawn helpers and command splitting utility
apps/engine/src/lib/engine.ts	Adds engine factory (setup/check/read/write/sync) + catalog builder
apps/engine/src/lib/destination-test.ts	Adds test destination connector
apps/engine/src/lib/destination-exec.ts	Adds subprocess Destination wrapper
apps/engine/src/index.ts	Adds app-level exports for engine package
apps/engine/src/cli/index.ts	Adds CLI entrypoint and global proxy agent patch
apps/engine/src/cli/command.ts	Adds CLI command wiring (OpenAPI-generated + hand-written serve)
apps/engine/src/api/index.ts	Adds API server entrypoint
apps/engine/src/tests/sync.test.ts	Adds engine lifecycle test using Docker Postgres
apps/engine/src/tests/docker.test.ts	Adds Docker image build/run tests with timeouts
apps/engine/package.json	Adds engine package metadata, exports, deps (incl. https-proxy-agent)
TODOs.md	Adds repository TODO tracker
README.md	Removes prior top-level README (legacy content)
Dockerfile	Reworks Docker build to use pnpm build + pnpm deploy artifact
AGENTS.md	Updates contributor guidance and monorepo layout notes
.vscode/settings.json	Updates Deno paths and Ruby LSP config
.vscode/extensions.json	Adds VS Code extension recommendation
.verdaccio/config.yaml	Adds Verdaccio config for local registry
.ruby-version	Pins Ruby version
.prettierignore	Ignores docs/out and terraform artifacts
.npmrc	Routes @stripe packages to registry via STRIPE_NPM_REGISTRY
.github/workflows/release.yml	Reworks release flow to promote by SHA to Docker Hub and npmjs
.github/workflows/docs.yml	Replaces MkDocs deploy with Vercel deploy and Slidev builds

Files not reviewed (1)

• docs/package-lock.json: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

This prepends sync_ to syncId, but sync IDs already appear to be formatted like sync_${string} in other parts of the repo. This will produce workflow IDs like sync_sync_abc123, breaking determinism and handle lookup. Consider returning syncId as-is, or only prefixing when the incoming ID is not already prefixed.

Suggested change

	return `sync_${syncId}`
	return syncId.startsWith('sync_') ? syncId : `sync_${syncId}`

[Copilot]

If this.configs.list() rejects, the promise rejection is currently unhandled (since only the inner handle.signal is caught). Make pushEvent async and await the list in a try/catch, or attach a .catch(...) to the list() promise to avoid unhandled rejections in production.

Suggested change

	this.configs.list().then((syncs) => {
	for (const sync of syncs) {
	if (sync.source.credential_id === credentialId) {
	const handle = this.client.getHandle(this.workflowId(sync.id))
	handle.signal('stripe_event', event).catch(() => {
	// Workflow may not be running — ignore signal failures
	})
	}
	}
	})
	this.configs
	.list()
	.then((syncs) => {
	for (const sync of syncs) {
	if (sync.source.credential_id === credentialId) {
	const handle = this.client.getHandle(this.workflowId(sync.id))
	handle.signal('stripe_event', event).catch(() => {
	// Workflow may not be running — ignore signal failures
	})
	}
	}
	})
	.catch(() => {
	// Ignore config listing failures to avoid unhandled rejections
	})

[Copilot]

The emitted failure_type: 'auth_error' likely does not match the engine/protocol error schema (elsewhere in the repo, failure types are config_error | system_error | transient_error). Since createEngine.read() parses messages via Message.parse, this can cause runtime parse failures when the test source emits an error. Align failure_type to the protocol’s allowed values (e.g., use config_error for auth failures).

Suggested change

	failure_type: 'auth_error' as const,
	failure_type: 'config_error' as const,

[Copilot]

sync() calls setup() but never calls teardown(), so resources created during setup (e.g., webhook endpoints, connections, temp infra) may leak or remain allocated after a one-shot sync finishes or throws. Wrap the streaming portion in a try/finally and call this.teardown() in the finally block (and consider whether teardown should optionally skip shared-resource deletion).

Suggested change

	yield* pipe(this.write(this.read(input)), persistState(stateStore))
	try {
	yield* pipe(this.write(this.read(input)), persistState(stateStore))
	} finally {
	await this.teardown()
	}

[Copilot]

The blanket catch {} will also swallow real runtime errors thrown by the state-store package (e.g., migrations failing, missing env vars), silently falling back to noopStateStore() and potentially losing checkpoint persistence. Prefer checking for module-not-found errors specifically and rethrowing anything else.

Suggested change

	} catch {
	} catch (error) {
	const moduleName = `@stripe/sync-state-${params.destination_name}`
	const isModuleNotFoundError =
	error &&
	typeof error === 'object' &&
	'code' in error &&
	(error as { code?: unknown }).code === 'ERR_MODULE_NOT_FOUND' &&
	'message' in error &&
	typeof (error as { message?: unknown }).message === 'string' &&
	(error as { message: string }).message.includes(moduleName)
	if (!isModuleNotFoundError) {
	throw error
	}

[Copilot]

Building via git clone without pinning to a tag/commit makes the image non-reproducible and increases supply-chain risk (the build output changes depending on when CI runs). Pin the clone to a specific release tag or commit SHA (and ideally verify it), so proxy behavior is deterministic across environments.

Suggested change

	RUN git clone --depth 1 https://github.com/stripe/smokescreen.git .
	ARG SMOKESCREEN_VERSION=v0.0.0
	RUN git clone --depth 1 --branch ${SMOKESCREEN_VERSION} https://github.com/stripe/smokescreen.git .

[Copilot]

Patching https.globalAgent globally affects all HTTPS traffic in the process and does not account for common proxy controls like NO_PROXY/no_proxy (e.g., bypassing proxy for localhost/internal services). Consider respecting NO_PROXY or scoping proxy usage to the specific HTTP clients that need it to avoid unexpected routing changes in long-running serve mode.

Suggested change

	import { HttpsProxyAgent } from 'https-proxy-agent'
	const proxyUrl = process.env.HTTPS_PROXY || process.env.HTTP_PROXY
	if (proxyUrl) {
	// Patch the global HTTPS agent so gaxios/googleapis routes through the proxy
	https.globalAgent = new HttpsProxyAgent(proxyUrl)
	}
	import { ProxyAgent } from 'proxy-agent'
	// Use a ProxyAgent that respects standard proxy env vars, including NO_PROXY/no_proxy.
	// This ensures gaxios/googleapis and other HTTPS clients route through the proxy when
	// configured, while bypassing it for hosts covered by NO_PROXY.
	const proxyAgent = new ProxyAgent()
	https.globalAgent = proxyAgent as unknown as typeof https.globalAgent

[Copilot]

SourceConfig in sync-types.ts requires livemode: boolean for stripe-api-core and stripe-api-reporting, but this example omits it. Because this file uses satisfies Sync, it won’t type-check as written. Add the required livemode field (and similarly update the other example sync).

[Copilot]

The PR title/description focuses on Smokescreen proxy support, but this diff also introduces a new Next.js visualizer app (plus extensive docs and new engine/service packages). Either update the PR description to cover these additional changes, or split unrelated changes into separate PRs to keep review scope and rollout risk manageable.

[tonyxiao]

Closing in favor of #187