Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -286,9 +286,19 @@ jobs:
echo "::error::dbt-tools bundle contains hardcoded absolute path in __dirname"
exit 1
fi
# Verify __dirname was patched to runtime resolution
if ! grep -q 'import.meta.dirname' packages/dbt-tools/dist/index.js; then
echo "::error::dbt-tools bundle missing import.meta.dirname patch"
# The bridge path is built at runtime from the bundle's own URL
if ! grep -q 'fileURLToPath(import.meta.url)' packages/dbt-tools/dist/index.js; then

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor, non-blocking (the preceding bun run build step already fails on this): the release smoke check is weaker than copy-python.ts.

Mutation on the built bundle: replacing x.join(_, "node_python_bridge.py") with a literal /home/runner/work/…/node_python_bridge.py still passes all three release greps. fileURLToPath(import.meta.url) is still present, and there is no var __dirname = "/…". copy-python.ts (its "node_python_bridge.py" include) and build-integrity (its checkout-path check) both catch it. To make this step stand on its own, also grep for "node_python_bridge.py" and for $GITHUB_WORKSPACE in dist/index.js.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in c0f6a99. The step now checks two more things in dist/index.js:

  • the bare "node_python_bridge.py" literal must be present;
  • $GITHUB_WORKSPACE must not appear. The step also fails if the variable is unset, because an empty pattern would match everything.

Results on a local build:

  • a clean build passes;
  • your mutation (the join replaced by a literal checkout path) fails on the first new check;
  • a checkout path appended to the bundle fails on the second.

echo "::error::dbt-tools bundle lacks fileURLToPath(import.meta.url), which the bridge path is built from"
exit 1
fi
# ...joined with the bare script name, not a literal path ending in it
if ! grep -qF '"node_python_bridge.py"' packages/dbt-tools/dist/index.js; then
echo "::error::dbt-tools bundle lacks the \"node_python_bridge.py\" name the bridge path is joined from"
exit 1
fi
# No path from this checkout baked into the bundle
if [ -z "$GITHUB_WORKSPACE" ] || grep -qF "$GITHUB_WORKSPACE" packages/dbt-tools/dist/index.js; then
echo "::error::dbt-tools bundle contains the build checkout path, or GITHUB_WORKSPACE is unset"
exit 1
fi
echo "dbt-tools smoke test passed"
Expand Down
82 changes: 18 additions & 64 deletions bun.lock

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions packages/dbt-tools/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@
"scripts": {
"build": "bun build src/index.ts --outdir dist --target node --format esm && bun run script/copy-python.ts",
"typecheck": "tsc --noEmit",
"test": "bun test test/cli.test.ts test/config.test.ts test/dbt-cli.test.ts test/dbt-resolve.test.ts test/build-integrity.test.ts --timeout 30000",
"test": "bun test test/cli.test.ts test/config.test.ts test/dbt-cli.test.ts test/dbt-resolve.test.ts test/adapter-bun.test.ts test/build-integrity.test.ts --timeout 30000",
"test:e2e": "bun test test/e2e/ --timeout 300000"
},
"dependencies": {
"@altimateai/dbt-integration": "^0.2.2"
"@altimateai/dbt-integration": "^0.3.15"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major: with the adapter now running, dbt's log output goes to the process's stdout: over the TUI and into run --format json.

0.3.15's createPythonBridge spawns Python with the default stdio: ["pipe", process.stdout, process.stderr, "pipe"]. 0.2.x had the same default, but under Bun it never got that far. On a compiled binary, the stdout of debug agent build --tool sql_execute begins with dbt's Registered adapter: duckdb=… and Unable to do partial parsing… lines. Then come two NodeParser Error reading dbt_loom.config.yml … ENOENT dumps with source frames, from the library's console.debug, which fires for every project without a dbt_loom config. Nothing went to stderr.

  • TUI: the server runs in a Bun Worker. worker-console-guard.ts overrides process.stdout.write and console.*, but a child that inherits fd 1 bypasses that. Repro: a Worker that no-ops process.stdout.write and spawns sh -c "echo X" with stdio: ["pipe", process.stdout, …] still prints X. So the first sql_execute in a dbt project draws dbt log lines over the TUI.
  • Headless: run --format json writes JSON lines to stdout (run.ts:735), and these lines get mixed in.

DBT_QUIET=true does not suppress them, and 0.3.16 creates the bridge the same way. Suggested fix: give the bridge piped stdout/stderr and route them to the DBTTerminal or log buffer. That needs an option in dbt-integration. Also add an output-hygiene check on the dbt-first path.

@ralphstodomingo ralphstodomingo Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The fix belongs in @altimateai/dbt-integration, and it's in review there for 0.3.17 (moved up from 0.3.18):

  • The bridge's child gets piped stdout and stderr, read line by line into the host's DBTTerminal at debug level. It's the default rather than an option, so every host gets it. dbt-tools' terminal drops debug, so here those lines go nowhere.
  • The dbt_loom read logs nothing when there's no config (those were the NodeParser Error reading dbt_loom.config.yml dumps), and the remaining console.log calls go through the terminal too.

Checked under Bun 1.3.14 with dbt-tools' adapter on jaffle (create(), then immediatelyExecuteSQL): stdout goes from 63 lines to 0 in core mode and from 136 to 0 in corecommand, and the query still returns its rows.

This PR moves to that release once it's out and past the 3-day minimumReleaseAge; we're not adding a minimumReleaseAgeExcludes entry for it. The bump will add the output-hygiene check you asked for on the dbt-first path.

},
"devDependencies": {
"@tsconfig/bun": "catalog:",
Expand Down
31 changes: 13 additions & 18 deletions packages/dbt-tools/script/copy-python.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { cpSync, existsSync, readFileSync, writeFileSync } from "fs"
import { cpSync, existsSync, readFileSync } from "fs"
import { dirname, join } from "path"

const dist = join(import.meta.dir, "..", "dist")
Expand All @@ -20,23 +20,18 @@ if (!existsSync(bridgePy)) {
cpSync(bridgePy, join(dist, "node_python_bridge.py"))
console.log(`Copied node_python_bridge.py → dist/`)

// 3. Fix the hardcoded __dirname that bun bakes at compile time.
// Replace it with a runtime resolution so the bridge script is found
// relative to the built index.js, not the CI runner's node_modules.
// 3. dbt-integration resolves node_python_bridge.py at runtime from its own
// location (`fileURLToPath(import.meta.url)`), which in this bundle is dist/,
// where step 2 put the script. This only checks that the bundle still has
// the references that path is built from and no path baked in at build time;
// test/build-integrity.test.ts runs the packaged CLI from another directory
// to check where the script is found.
const indexPath = join(dist, "index.js")
let code = readFileSync(indexPath, "utf8")
const pattern = /var __dirname\s*=\s*"[^"]*python-bridge[^"]*"/
if (pattern.test(code)) {
// import.meta.dirname is supported by Bun and Node >= 20.11.0.
// Fallback via __require handles older runtimes where import.meta.dirname is unavailable.
const replacement = `var __dirname = typeof import.meta.dirname === "string" ? import.meta.dirname : __require("path").dirname(__require("url").fileURLToPath(import.meta.url))`
code = code.replace(pattern, replacement)
writeFileSync(indexPath, code)
console.log(`Patched __dirname in dist/index.js`)
} else {
const found = code.match(/var __dirname[^;]*/)?.[0] ?? "(not found)"
console.error(`ERROR: could not find python-bridge __dirname to patch — the bundle format may have changed`)
console.error(` Pattern: ${pattern}`)
console.error(` Nearest match: ${found}`)
const code = readFileSync(indexPath, "utf8")
const baked = /var __dirname\s*=\s*"(?:[A-Za-z]:\\\\|\/)/
if (!code.includes("fileURLToPath(import.meta.url)") || !code.includes(`"node_python_bridge.py"`) || baked.test(code)) {
Comment thread
ralphstodomingo marked this conversation as resolved.
Resolved
console.error(`ERROR: dist/index.js lacks the references the bridge path is built from, or has a build-time path`)
console.error(` Has the @altimateai/dbt-integration bundle format changed?`)
process.exit(1)
}
console.log(`Checked dist/index.js: bridge path references present, no build-time path`)
20 changes: 7 additions & 13 deletions packages/dbt-tools/src/adapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ import {
GraphParser,
SourceParser,
TestParser,
UnitTestParser,
SemanticModelParser,
ExposureParser,
FunctionParser,
DocParser,
Expand Down Expand Up @@ -141,6 +143,8 @@ export async function create(cfg: Config): Promise<DBTProjectIntegrationAdapter>
defer,
changed,
cloudVariantDetector,
// Every variant but fusion-stable runs on a dbt Core command engine, which needs these.
{ pythonDBTCommandExecutionStrategy: python, dbtConfiguration: config, dbtIntegrationClient: client },
)

const command = (
Expand Down Expand Up @@ -170,19 +174,7 @@ export async function create(cfg: Config): Promise<DBTProjectIntegrationAdapter>
defer: DeferConfig,
changed: () => void,
): DBTProjectIntegration =>
new DBTFusionCommandProjectIntegration(
infra,
factory,
cli,
runtime,
provider,
term,
root,
diag,
defer,
changed,
cloudVariantDetector,
)
new DBTFusionCommandProjectIntegration(infra, factory, cli, runtime, provider, term, root, diag, defer, changed)

const adapter = new DBTProjectIntegrationAdapter(
config,
Expand All @@ -200,11 +192,13 @@ export async function create(cfg: Config): Promise<DBTProjectIntegrationAdapter>
new GraphParser(term),
new SourceParser(term),
new TestParser(term),
new UnitTestParser(term),
new ExposureParser(term),
new FunctionParser(term),
new DocParser(term),
term,
new ModelDepthParser(term, client, config),
new SemanticModelParser(term),
)

await adapter.initialize()
Expand Down
10 changes: 2 additions & 8 deletions packages/dbt-tools/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,6 @@ function flag(args: string[], name: string): string | undefined {

function diagnose(err: Error): { error: string; fix?: string } {
const msg = err.message || String(err)
if (msg.includes("private field") || msg.includes("#stdin") || msg.includes("#stdout")) {
return {
error: "Internal error: Bun runtime incompatibility with python-bridge. Please report this issue.",
fix: "Try updating: bun install, or file a bug at https://github.com/AltimateAI/altimate-code/issues",
}
}
if (msg.includes("IPC channel") || msg.includes("ERR_IPC_CHANNEL_CLOSED")) {
return {
error: "Failed to start Python bridge. The dbt Python process exited unexpectedly.",
Expand Down Expand Up @@ -109,7 +103,7 @@ function bail(err: unknown): never {
process.exit(1)
}

// Catch unhandled rejections (bluebird throws outside normal promise chains)
// An error that escapes main() still ends as a JSON error on stdout
process.on("unhandledRejection", bail)
process.on("uncaughtException", bail)

Expand Down Expand Up @@ -142,7 +136,7 @@ async function main() {
const { configure } = await import("./dbt-cli")
configure({ pythonPath: cfg.pythonPath, projectRoot: cfg.projectRoot })

// Lazy import to avoid loading python-bridge until needed
// Lazy import: the adapter loads dbt-integration, which commands above don't need
let adapter
try {
const { create } = await import("./adapter")
Expand Down
50 changes: 50 additions & 0 deletions packages/dbt-tools/test/adapter-bun.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
import { describe, test, expect } from "bun:test"
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "fs"
import { tmpdir } from "os"
import { dirname, join } from "path"
import { $ } from "bun"
import { standInPython } from "./stand-in-python"

const pkg = join(import.meta.dir, "..")

// The published-package tests run the bundle under Node. The TUI and the compiled
// binary run the adapter under Bun, the runtime the bridge used to fail on, so this
// runs the CLI from source under Bun. HOME is a temp dir from the start, so no
// ~/.dbt/dbt_cloud.yml on this machine picks dbt Cloud mode's variant.
describe.skipIf(process.platform === "win32")("the adapter under Bun", () => {
const script = join(dirname(require.resolve("@altimateai/dbt-integration")), "node_python_bridge.py")

// The stand-in fails the bridge run. The dbt Core modes then exit 1 with the bridge's
// report of that exit; `info` in dbt Cloud mode still answers from the project. A
// failure before the spawn shows its own message in the comparison.
test.each([
["core", { exitCode: 1, output: expect.stringContaining("Python process closed with exit code 3") }],
["corecommand", { exitCode: 1, output: expect.stringContaining("Python process closed with exit code 3") }],
["cloud", { exitCode: 0 }],
])("%s mode spawns the bridge script", async (dbtIntegration, expected) => {
const root = mkdtempSync(join(tmpdir(), "dbt-tools-bun-"))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Neutralise HOME here too.

startPublished in build-integrity.test.ts points HOME at the temp dir specifically so that no ~/.dbt/dbt_cloud.yml exists (see its comment at line 83). This test calls create() against the real home, so the cloud case's variant detection and credentials come from the developer's machine: with a dbt Cloud config present the variant can resolve differently (e.g. fusion-stable, which never spawns the bridge), so the case passes in CI (no ~/.dbt) and can fail locally. Set HOME to a temp dir for the duration, as the published-package test does.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 5018286. With this machine's ~/.dbt/dbt_cloud.yml, the cloud case timed out at 5 s.

Pointing HOME at the temp dir inside the test process wasn't enough:

  • Bun's os.homedir() doesn't follow a runtime change to HOME.
  • The detector reads the home directory once, in its constructor, through import * as os, so a spy can't reach it either.

So the test now runs the CLI from source as a Bun subprocess (process.execPath src/index.ts info), with HOME set to the temp dir from the start, as the published-package test does. Results:

  • It passes with both the real home and a temp home.
  • It fails on 0.2.14 in all three modes.
  • With the Cloud engine dependencies removed, it fails with "engine not initialized".

try {
const { python, runs } = standInPython(root)
const project = join(root, "project")
mkdirSync(project)
writeFileSync(join(project, "dbt_project.yml"), "name: p\nversion: '1.0'\nprofile: p\n")
const home = join(root, "home")
mkdirSync(join(home, ".altimate-code"), { recursive: true })
writeFileSync(
join(home, ".altimate-code", "dbt.json"),
JSON.stringify({ projectRoot: project, pythonPath: python, dbtIntegration, queryLimit: 500 }),
)

const cli = await $`${process.execPath} ${join(pkg, "src/index.ts")} info`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: This duplicates the CLI harness in build-integrity.test.ts.

startPublished() (build-integrity.test.ts:70-100) already does this exact sequence: write a temp <home>/.altimate-code/dbt.json, start the stand-in python, run the CLI with HOME redirected, then compare captured exitCode/stdout and assert the spawned script. The only differences here are the runtime (process.execPath vs node) and the entry point (src/index.ts vs the installed dist/index.js). Extracting one helper next to standInPython (e.g. runDbtCli(root, { command, entry })) and calling it from both files would remove ~20 duplicated lines and stop the two layouts from drifting apart when the config shape or assertions change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Holding this one for the dependency bump that will follow this PR. That bump changes both harnesses:

  • They need a stand-in dbt that answers --version, because the next dbt-integration checks for the dbt Cloud CLI before it creates a Cloud engine.
  • They also get the output-hygiene check from the review.
    Extracting a shared runDbtCli helper then, next to standInPython, means the two layouts change once. For now, the shared part is the stand-in, which already lives in one place.

.cwd(root)
.env({ ...process.env, HOME: home })
.quiet()
.nothrow()

expect({ exitCode: cli.exitCode, output: cli.stdout.toString() }).toMatchObject(expected)
expect(runs()).toContain(script)
} finally {
rmSync(root, { recursive: true, force: true })
}
})
})
Loading
Loading