Ci/build once reuse binaries by vsilent · Pull Request #268 · trydirect/stacker

vsilent · 2026-09-23T13:11:14Z

This pull request introduces significant improvements to the build and deployment pipeline, focusing on optimizing Docker builds, improving CI efficiency, and enhancing developer tooling and documentation. The changes streamline how binaries are built and included in Docker images, reduce redundant compilation, and improve caching strategies. Additionally, it introduces a new pre-commit hook for secret detection and updates documentation and versioning.

Build & CI Pipeline Optimization:

The CI workflow (.github/workflows/docker.yml) now builds Rust binaries inside a rust:bookworm container, ensuring glibc compatibility between build and runtime environments and preventing runtime errors due to version mismatches.
Instead of compiling binaries multiple times, the workflow produces release binaries once and passes them as artifacts to the Docker build, which can now accept prebuilt binaries or build from source based on a build argument (BINARIES). This greatly reduces build times and resource usage. [1] [2] [3] [4] [5] [6]
The Dockerfile is refactored to support two build paths: using prebuilt binaries (from CI) or building them inside the image, with a new config stage for shared config files and tools. [1] [2] [3]
The caching strategy is improved by switching to Swatinem/rust-cache@v2 for only the Cargo registry and git, avoiding caching the target directory to prevent unbounded disk usage. [1] [2]

Developer Tooling & Quality:

A new pre-commit hook (check-staged-secrets) is added to prevent committing concrete secret values, improving security practices.

Documentation & Versioning:

The CHANGELOG.md is updated for version 0.3.3, detailing new features, security improvements, and bug fixes. [1] [2]
The project version is bumped to 0.3.3 in Cargo.toml.
Documentation is improved to clarify the secret lifecycle and details about the baked snapshot process.

Summary of Most Important Changes:

Build and CI Pipeline Improvements

Rust binaries are now built in a rust:bookworm container to ensure runtime compatibility, and the workflow passes prebuilt binaries to Docker, eliminating redundant compilation and reducing CI times. [1] [2] [3] [4] [5] [6] [7]
The Dockerfile supports a new build argument (BINARIES) to select between using prebuilt binaries or building from source, with a new config stage for shared files and tools. [1] [2] [3]
The caching strategy is updated to avoid caching the target directory, preventing disk space issues in CI. [1] [2]

Developer Tooling

Added a pre-commit hook to check for staged secrets, preventing accidental leaks of sensitive information in commits.

Documentation and Versioning

Updated CHANGELOG.md with detailed notes for version 0.3.3, including new features, fixes, and improvements. [1] [2]
Improved documentation on secret lifecycle and snapshot process for clarity and compliance.
Bumped project version to 0.3.3 in Cargo.toml.

- Chat session management with archive and encryption - Agent hardening: per-tenant ownership, token digest verification, fail-closed auth - Marketplace field policy: config_contract, generated-field stripping, derived_jwt signing - Project sync, one-click deploy improvements, deployment container tracking - SSH key authorization fixes, mTLS for Vault, port validation - Stale project/server cleanup, audit-log cron, env size validator - Multiple BDD and migration fixes

- create_handler now uses update_metadata_for_resubmit for submitted/under_review/approved templates - CLI submit command uses resubmit endpoint for approved templates instead of submit endpoint - adds marketplace_resubmit client method for POST /api/templates/{id}/resubmit

…lookup for resubmit - build_project_app now copies config_contract from the form app - get_source_project_id checks all versions (not just latest) since resubmit_with_new_version creates a new version row before set_source_project_id is called

…ubmit - unit test: project_level_apps_from_form propagates config_contract - integration test: sync persists config_contract on project apps - integration test: create_handler updates approved template metadata - integration test: resubmit with new version preserves source_project_id

The insert/update SQL does not include config_contract — it is persisted via a dedicated set_config_contract call. sync_project_level_apps_from_form now calls set_config_contract after each insert/update when the form app declares a config_contract.

Remove dead agent rows whose deployment is deleted/missing and that show no sign of life within 30 days (last_heartbeat AND audit_log). Remove rows with structurally invalid deployment_hash unconditionally — these can never authenticate and often leak a raw token in plaintext. The audit_log check protects agents that are alive but failing authentication: last_heartbeat only advances on successful wait/report, while audit_log captures auth_failure entries. Migration 20260113000002 already converted audit_log.created_at to timestamptz — no new migration needed. Includes 9 integration tests covering the key cases from the sweep plan.

Feature/agent sweep

…ed snapshots - Add parameterize_compose_env_vars() to replace literal env values with ${VAR} references in generated compose files - Integrate into deploy pipeline so compose never contains author secrets - Docker Compose resolves ${VAR} from co-located .env at runtime - Add 3 unit tests for parameterization behavior This fixes the security issue where every buyer of a marketplace template received the author's literal secrets in the baked compose file.

feat(compose): parameterize env vars to prevent secret leakage in bak…

prepare server for baking, clean creds, keys, logs etc

Follow-up to 27017fa. An adversarial review of that change found the sanitize step could destroy the build box, refuse healthy deploys, or report success over an image it had not actually checked. Volume reset was enumerating every volume on the host, not the project's. On a real build box that removes the ingress' certificates and the agent's state, and aborts the bake on the first volume still held by a running container. Scoped to the project's own compose and matched through Compose's own label; keep entries are validated and matched on whole segments. The required-key list was read from the text of the compose file, so `${VAR:-default}` and `$$`-escaped text counted as required and no buyer could ever satisfy them. It now comes from the contract — the only thing that has a source on a buyer's machine. References the contract does not cover are resolved back to their literals before the snapshot, since a buyer's env file is replaced wholesale and would leave them empty. Further: - config_contract is now the only authority on what is sensitive; the weaker name heuristic is gone. A credential embedded inside a larger value (a password inside a DSN) is cleared in .env as well as in compose. - The author's own access no longer survives: authorized_keys, private keys, known_hosts and registry credentials are removed alongside the machine identity. Cloud-init appends the buyer's key rather than replacing the file, so a key left behind would grant root on every server cloned from the image. - The tear-down runs before the rewrites, so a cleared .env cannot fail `docker compose down` with the files already modified. - Files are written in chunks, so a large compose no longer exceeds the argument-length limit mid-sanitize; error messages no longer echo the payload. - A failure reports which steps completed and whether a retry against the same box is still equivalent. - The bake refuses when no contract resolved, when a compose reads values through `env_file:` that a buyer would silently lose, on an unknown argument, and when the contract describes a different version. - Value-stripping no longer clears the field policies inside config_contract itself. Adds scripts/check-staged-secrets.sh, wired as a pre-commit check — the scanner already configured in .pre-commit-config.yaml was never installed, so nothing was checking. Adds docs/SECRET_LIFECYCLE.md, tracing where one value lives at each stage and which component owns it. 2083 unit tests and 329 BDD scenarios green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

fix(bake): close the gaps an audit found in build-box sanitization

Every healthcheck written the way the reference documents came out broken: /bin/sh: 1: CMD-SHELL: not found Docker wraps a plain string `test` in `CMD-SHELL` itself, so a string that already spells the prefix out gets wrapped twice and the container tries to execute a program named `CMD-SHELL`. The prefix only means anything in the list form. The generator was emitting the author's string verbatim. Observed on a real deploy: both stackpilot services that declare a healthcheck came up unhealthy while the application itself was fine. Harmless on its own, but the state is frozen into a baked snapshot, and a stack using `depends_on: condition: service_healthy` would never start. The decision now lives in one place. `compose_service_sync` already had `healthcheck_test_value` doing this correctly for the server-side path; the generator delegates to it and only renders the result as inline YAML. That also picks up a subtlety a second implementation would have missed: `CMD` executes argv directly, so a command containing `&&`, `|`, `$` or redirection is emitted as `CMD-SHELL` instead. An explicit list written by the author now passes through untouched. The reference said `test: "CMD pg_isready -U postgres"` and left it there. It now documents all three accepted forms, what each one runs, and why the prefix works here but not in a plain compose file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

fix(compose): stop double-wrapping healthcheck test commands

The build filled the runner's disk and died in post-job cleanup: Unhandled exception. System.IO.IOException: No space left on device Every check had passed — cargo check, the test suite, the BDD suite, rustfmt, all four release binaries. The job only failed while saving its cache, after 33 minutes. `actions/cache` was storing the whole `target` directory, and `restore-keys: docker-` meant each run started from an older, already bloated cache, added to it, and saved a larger one — so every run raised the next one's floor. For a workspace building five binaries in both debug and release that ratchets past the ~14 GB a runner has free. Replaced with Swatinem/rust-cache, caching registry and git only. rust.yml already made this exact call, with a comment explaining why; the two workflows now follow one rule. The trade-off is a slower docker workflow, since compilation output is no longer reused between runs. That is the right side to err on: a slow pipeline is an inconvenience, a pipeline that cannot finish is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

ci: stop caching the Rust target directory in the docker workflow

The bake drops a stack's volumes so a buyer's machine initialises them with its own credentials. Some must survive that: Ollama weights are gigabytes behind a 600s pull, Qdrant embeddings cost as much to recompute — preserving them is why the snapshot exists. Until now the exceptions were a `match` on the stack slug inside `bake_finalize.rs`. Four stacks were listed, floci and the rest were queued, and each one needed a code change and a rebuilt binary. The author knows which of their volumes are expensive and which hold credentials; the platform does not. So it goes in `config_contract`, beside the fields, reusing everything already built for them: the `mutability` vocabulary, and the path from stacker.yml through the submit body to `stack_template_version.config_contract` and on to `bake.rs`, which resolves it before finalize runs. config_contract: services: stackpilot-ollama: volumes: stackpilot_ollama: { mutability: fixed } `fixed` ships the content inside the image; `generated` — the default for anything undeclared — drops it. Erring that way costs a rebuild; the opposite default would hand the author's credentials to every buyer. `provided` and `editable` describe who types a value and are rejected: a volume holds state, not a value. The platform validates only the name, which is interpolated into a shell pattern. It deliberately does not second-guess the declaration. An earlier revision of this change did. It refused any volume whose service declares `generated` or `provided` fields, reasoning that such a service persists the secret. Measuring real containers killed that rule: a Postgres data directory holds `SCRAM-SHA-256$4096:…` and not the password in any searchable form; n8n keeps its own encryption key inside `database.sqlite`; a Qdrant volume holds only collections, because Qdrant reads its API key from the environment at every start. The secret is absent from all three — so neither the field-based rule nor a search of the volume's bytes tells the two that must reset from the one that must be kept. The difference is behavioural, and only the author can see it. Left in, the rule would have forced ai-knowledge-base to recompute its embeddings on every buyer's machine: the exact expense a snapshot avoids. FinalizeContext now carries the parsed contract rather than a flattened key set, since the kind-per-service structure is what the volume policy needs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

feat(contract): let the author declare which volumes survive a bake

A run takes 47 minutes, and the release binaries are compiled twice in it: once by the test job, then again from scratch inside the Docker build. The first set was already being packed into `app.tar.gz` and uploaded — nothing ever downloaded it. The reason it could not simply be reused is the one that has to be got right: the binaries are dynamically linked against glibc, and glibc is forward- but not backward-compatible. Built on the runner (Ubuntu 24.04, glibc 2.39) they would not start on debian:bookworm-slim (glibc 2.36). So the test job now runs inside `rust:bookworm` — the same image the Dockerfile builds in, and the one the runtime stage is derived from. The environments match exactly, and the Docker job copies the artifact instead of recompiling. The Dockerfile keeps both paths. `BINARIES=prebuilt` takes them from a build context; the default still compiles from source, so a local `docker build` works unchanged. Both were verified to parse, and the prebuilt path was built end to end: the four binaries, the config files and the sqlx CLI all land in the image with no compilation. Two things fell out along the way. The job now builds all four binaries the image needs — `console` and `backfill_field_policy` were missing from the artifact, which is part of why it could not be used. And `cargo install sqlx-cli` moved to its own small stage with only the postgres and rustls features, so the prebuilt path no longer pays 110 seconds of it to fetch two YAML files. Inside a job container, service containers resolve by name rather than 127.0.0.1, so PGHOST changes accordingly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The reference documented `required_env:` under `config_contract`. No such field exists — the parser knows `required`, and since every contract type denies unknown fields, anyone following the example would have had their submission rejected. Removing it was right. It took the whole section with it, though, leaving the reference with no description of field policy at all: only the volume subsection added last week remained. An author reading this file would not learn that `mutability` exists. The design document in config/docs covers it, but that is not where someone writing a stacker.yml looks. Restored with the four mutabilities, the keys that apply to each, the legacy three-list shorthand, and a note that publishing is refused until secret-shaped fields carry a policy. Every example here was run through the parser, including the removed `required_env`, which is confirmed to be rejected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

gitguardian · 2026-09-23T13:12:12Z

⚠️ GitGuardian has uncovered 6 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
37456823	Triggered	Generic Password	`27017fa`	src/helpers/bake_finalize.rs	View secret
37449298	Triggered	Generic Password	`4b1e286`	src/cli/generator/compose.rs	View secret
37449299	Triggered	Generic High Entropy Secret	`4b1e286`	src/cli/generator/compose.rs	View secret
37456824	Triggered	Generic High Entropy Secret	`13a65d3`	src/cli/generator/compose.rs	View secret
37490585	Triggered	Generic High Entropy Secret	`51d5ff1`	src/cli/generator/compose.rs	View secret
37490586	Triggered	Generic Password	`51d5ff1`	src/cli/generator/compose.rs	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

+
+        assert!(
+            result.contains("postgresql://stackpilot:${POSTGRES_PASSWORD}@db:5432/stackpilot"),
+            "password replaced in place:\n{result}"


+        );
+        assert!(
+            !result.contains("0123456789abcdef0123456789abcdef"),
+            "no literal left:\n{result}"


+            .expect("no conflict");
+        assert!(
+            result.contains("GREETING: administrator"),
+            "untouched:\n{result}"


+                .expect("no conflict");
+        assert!(
+            result.contains("aaaaaaaaaaaaaaaa"),
+            "left literal:\n{result}"


+
+        assert!(
+            result.contains("image: myapp:0123456789abcdef0123456789abcdef"),
+            "image digest untouched:\n{result}"


+        );
+        assert!(
+            result.contains("TOKEN: ${TOKEN}"),
+            "env replaced:\n{result}"


The run failed on `sudo: not found`. The container runs as root and does not ship sudo, and it does not need to: `rust:bookworm` already has pkg-config, libssl-dev and a C toolchain, and protoc never comes from the system — `build.rs` points PROTOC at a vendored binary unless one is set. Verified against the image. The step is gone. The error that was actually reported was `no such command: nextest`, which is not what went wrong. Both test steps carried `if: always()`, so they ran after the setup step failed and the nextest install had been skipped. Dropped on the first, narrowed to `success() || failure()` on the second, which is what was wanted: run both suites even if one fails, without reporting on an environment that was never built. Two more, found while looking rather than by the next 47-minute run: `--features explain` applies to the whole `cargo build`, not to the `--bin` it follows, so folding four binaries into one invocation was quietly shipping `server` with explain-logging on — a different binary from the one the image has always carried. Split in two; only `console` and the re-featured casbin dependency recompile. `.dockerignore` is empty, so the unpacked binaries and `app.tar.gz` were being sent to buildkit as part of `context: .` — hundreds of megabytes, twice, eating back the time this change exists to save. They now unpack to `runner.temp`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

robotizeit and others added 26 commits September 18, 2026 08:22

logging, db cronjob migration fix, reclaim disk space query / VACUUM

098c80f

PIPE audit log, not implemented, on hold, should be discussed

b1df4cf

test: authenticate audit BDD ingest as a registered agent

b04fbe0

fix: include config_contract in sync payload

0515e96

Merge pull request #258 from trydirect/feature/agent-sweep

6b7f4a8

Feature/agent sweep

fix(compose): cover protected service environment keys

1a10914

prepare server for baking, clean creds, keys, logs etc

27017fa

Merge pull request #259 from trydirect/feature/agent-sweep

4b1e286

feat(compose): parameterize env vars to prevent secret leakage in bak…

Merge pull request #261 from trydirect/bake-improvement

13a65d3

prepare server for baking, clean creds, keys, logs etc

Merge pull request #263 from trydirect/feature/bake-sanitization

51d5ff1

fix(bake): close the gaps an audit found in build-box sanitization

Merge pull request #264 from trydirect/fix/healthcheck-test-form

2623fed

fix(compose): stop double-wrapping healthcheck test commands

Merge pull request #265 from trydirect/ci/no-target-cache

996cdca

ci: stop caching the Rust target directory in the docker workflow

Merge pull request #266 from trydirect/feat/contract-volume-policy

e34b2dc

feat(contract): let the author declare which volumes survive a bake

github-advanced-security Bot found potential problems Sep 23, 2026

View reviewed changes

Navigation Menu

Ci/build once reuse binaries - #268

Ci/build once reuse binaries#268
vsilent wants to merge 27 commits into
mainfrom
ci/build-once-reuse-binaries

vsilent commented Sep 23, 2026

Uh oh!

gitguardian Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

vsilent commented Sep 23, 2026

Uh oh!

gitguardian Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ GitGuardian has uncovered 6 secrets following the scan of your pull request.

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gitguardian Bot commented Sep 23, 2026 •

edited

Loading