Conversation
- Chat session management with archive and encryption - Agent hardening: per-tenant ownership, token digest verification, fail-closed auth - Marketplace field policy: config_contract, generated-field stripping, derived_jwt signing - Project sync, one-click deploy improvements, deployment container tracking - SSH key authorization fixes, mTLS for Vault, port validation - Stale project/server cleanup, audit-log cron, env size validator - Multiple BDD and migration fixes
- create_handler now uses update_metadata_for_resubmit for submitted/under_review/approved templates
- CLI submit command uses resubmit endpoint for approved templates instead of submit endpoint
- adds marketplace_resubmit client method for POST /api/templates/{id}/resubmit
…lookup for resubmit - build_project_app now copies config_contract from the form app - get_source_project_id checks all versions (not just latest) since resubmit_with_new_version creates a new version row before set_source_project_id is called
…ubmit - unit test: project_level_apps_from_form propagates config_contract - integration test: sync persists config_contract on project apps - integration test: create_handler updates approved template metadata - integration test: resubmit with new version preserves source_project_id
The insert/update SQL does not include config_contract — it is persisted via a dedicated set_config_contract call. sync_project_level_apps_from_form now calls set_config_contract after each insert/update when the form app declares a config_contract.
Remove dead agent rows whose deployment is deleted/missing and that show no sign of life within 30 days (last_heartbeat AND audit_log). Remove rows with structurally invalid deployment_hash unconditionally — these can never authenticate and often leak a raw token in plaintext. The audit_log check protects agents that are alive but failing authentication: last_heartbeat only advances on successful wait/report, while audit_log captures auth_failure entries. Migration 20260113000002 already converted audit_log.created_at to timestamptz — no new migration needed. Includes 9 integration tests covering the key cases from the sweep plan.
Feature/agent sweep
…ed snapshots
- Add parameterize_compose_env_vars() to replace literal env values with
${VAR} references in generated compose files
- Integrate into deploy pipeline so compose never contains author secrets
- Docker Compose resolves ${VAR} from co-located .env at runtime
- Add 3 unit tests for parameterization behavior
This fixes the security issue where every buyer of a marketplace template
received the author's literal secrets in the baked compose file.
feat(compose): parameterize env vars to prevent secret leakage in bak…
prepare server for baking, clean creds, keys, logs etc
Follow-up to 27017fa. An adversarial review of that change found the sanitize step could destroy the build box, refuse healthy deploys, or report success over an image it had not actually checked. Volume reset was enumerating every volume on the host, not the project's. On a real build box that removes the ingress' certificates and the agent's state, and aborts the bake on the first volume still held by a running container. Scoped to the project's own compose and matched through Compose's own label; keep entries are validated and matched on whole segments. The required-key list was read from the text of the compose file, so `${VAR:-default}` and `$$`-escaped text counted as required and no buyer could ever satisfy them. It now comes from the contract — the only thing that has a source on a buyer's machine. References the contract does not cover are resolved back to their literals before the snapshot, since a buyer's env file is replaced wholesale and would leave them empty. Further: - config_contract is now the only authority on what is sensitive; the weaker name heuristic is gone. A credential embedded inside a larger value (a password inside a DSN) is cleared in .env as well as in compose. - The author's own access no longer survives: authorized_keys, private keys, known_hosts and registry credentials are removed alongside the machine identity. Cloud-init appends the buyer's key rather than replacing the file, so a key left behind would grant root on every server cloned from the image. - The tear-down runs before the rewrites, so a cleared .env cannot fail `docker compose down` with the files already modified. - Files are written in chunks, so a large compose no longer exceeds the argument-length limit mid-sanitize; error messages no longer echo the payload. - A failure reports which steps completed and whether a retry against the same box is still equivalent. - The bake refuses when no contract resolved, when a compose reads values through `env_file:` that a buyer would silently lose, on an unknown argument, and when the contract describes a different version. - Value-stripping no longer clears the field policies inside config_contract itself. Adds scripts/check-staged-secrets.sh, wired as a pre-commit check — the scanner already configured in .pre-commit-config.yaml was never installed, so nothing was checking. Adds docs/SECRET_LIFECYCLE.md, tracing where one value lives at each stage and which component owns it. 2083 unit tests and 329 BDD scenarios green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(bake): close the gaps an audit found in build-box sanitization
Every healthcheck written the way the reference documents came out broken: /bin/sh: 1: CMD-SHELL: not found Docker wraps a plain string `test` in `CMD-SHELL` itself, so a string that already spells the prefix out gets wrapped twice and the container tries to execute a program named `CMD-SHELL`. The prefix only means anything in the list form. The generator was emitting the author's string verbatim. Observed on a real deploy: both stackpilot services that declare a healthcheck came up unhealthy while the application itself was fine. Harmless on its own, but the state is frozen into a baked snapshot, and a stack using `depends_on: condition: service_healthy` would never start. The decision now lives in one place. `compose_service_sync` already had `healthcheck_test_value` doing this correctly for the server-side path; the generator delegates to it and only renders the result as inline YAML. That also picks up a subtlety a second implementation would have missed: `CMD` executes argv directly, so a command containing `&&`, `|`, `$` or redirection is emitted as `CMD-SHELL` instead. An explicit list written by the author now passes through untouched. The reference said `test: "CMD pg_isready -U postgres"` and left it there. It now documents all three accepted forms, what each one runs, and why the prefix works here but not in a plain compose file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(compose): stop double-wrapping healthcheck test commands
The build filled the runner's disk and died in post-job cleanup: Unhandled exception. System.IO.IOException: No space left on device Every check had passed — cargo check, the test suite, the BDD suite, rustfmt, all four release binaries. The job only failed while saving its cache, after 33 minutes. `actions/cache` was storing the whole `target` directory, and `restore-keys: docker-` meant each run started from an older, already bloated cache, added to it, and saved a larger one — so every run raised the next one's floor. For a workspace building five binaries in both debug and release that ratchets past the ~14 GB a runner has free. Replaced with Swatinem/rust-cache, caching registry and git only. rust.yml already made this exact call, with a comment explaining why; the two workflows now follow one rule. The trade-off is a slower docker workflow, since compilation output is no longer reused between runs. That is the right side to err on: a slow pipeline is an inconvenience, a pipeline that cannot finish is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ci: stop caching the Rust target directory in the docker workflow
The bake drops a stack's volumes so a buyer's machine initialises them
with its own credentials. Some must survive that: Ollama weights are
gigabytes behind a 600s pull, Qdrant embeddings cost as much to recompute
— preserving them is why the snapshot exists.
Until now the exceptions were a `match` on the stack slug inside
`bake_finalize.rs`. Four stacks were listed, floci and the rest were
queued, and each one needed a code change and a rebuilt binary.
The author knows which of their volumes are expensive and which hold
credentials; the platform does not. So it goes in `config_contract`,
beside the fields, reusing everything already built for them: the
`mutability` vocabulary, and the path from stacker.yml through the
submit body to `stack_template_version.config_contract` and on to
`bake.rs`, which resolves it before finalize runs.
config_contract:
services:
stackpilot-ollama:
volumes:
stackpilot_ollama: { mutability: fixed }
`fixed` ships the content inside the image; `generated` — the default for
anything undeclared — drops it. Erring that way costs a rebuild; the
opposite default would hand the author's credentials to every buyer.
`provided` and `editable` describe who types a value and are rejected:
a volume holds state, not a value.
The platform validates only the name, which is interpolated into a shell
pattern. It deliberately does not second-guess the declaration.
An earlier revision of this change did. It refused any volume whose
service declares `generated` or `provided` fields, reasoning that such a
service persists the secret. Measuring real containers killed that rule:
a Postgres data directory holds `SCRAM-SHA-256$4096:…` and not the
password in any searchable form; n8n keeps its own encryption key inside
`database.sqlite`; a Qdrant volume holds only collections, because Qdrant
reads its API key from the environment at every start. The secret is
absent from all three — so neither the field-based rule nor a search of
the volume's bytes tells the two that must reset from the one that must
be kept. The difference is behavioural, and only the author can see it.
Left in, the rule would have forced ai-knowledge-base to recompute its
embeddings on every buyer's machine: the exact expense a snapshot avoids.
FinalizeContext now carries the parsed contract rather than a flattened
key set, since the kind-per-service structure is what the volume policy
needs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feat(contract): let the author declare which volumes survive a bake
9686684 added `volumes` to the contract's parser and to the struct, but `TargetConfigContract` has a hand-written `Serialize` that builds an intermediate struct field by field — and it did not know about the new one. So a declaration parsed correctly, then vanished on the way out. That path is `stacker submit`: the contract is serialized into the submit body. The declaration never reached the registry, and the bake would have reset the volume it was meant to keep. Nothing surfaced — the submit succeeded, the stored contract merely had an empty service block where the volumes should have been. Caught by querying the database after a real resubmit of stackpilot. Every existing test read a contract and asserted on the parsed result; none serialized one back. The new test round-trips through JSON and checks both directions, which is what the submit path actually does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The serialization defect fixed in the previous commit reached production
because every existing test parsed a contract and asserted on the result.
None serialized one back, so a kind missing from the hand-written
`Serialize` looked fine everywhere.
Two paths serialize it, and both were losing the declaration:
- `submit.rs:130` — the marketplace registry, covered by the round-trip
test added with the fix
- `stacker_client.rs:4102` — `stacker sync`, writing the contract onto
every app of a project. Covered here
A third reads it: `compose_env_keys` in `deploy.rs`. It takes fields and
must ignore the rest — a volume name is not an environment variable, and
parameterizing one would leave the compose asking for a value nothing
supplies. Covered here too.
Both new serialization tests were verified against the bug: with the fix
reverted they fail, with it applied they pass.
Two boundaries the audit surfaced also get tests. A legacy
`required`/`optional`/`secret` contract must round-trip untouched and must
not grow an empty `volumes:` key. And a service declaring only volumes and
no fields must survive serialization rather than collapsing to `{}` —
which is precisely the shape the production database showed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The worst of these would have published an image in a worse state than
before any of this work started, so they are grouped rather than split.
**The sanitize step could run against nothing and report success.**
Moving `FinalizeContext` to a parsed contract meant the flat protected-key
set was re-derived by serializing it back — and `Serialize` moves a
default-shaped generated secret out of `fields` into the legacy `secret:`
list, which `protected_keys_from_contract` did not read. For the most
ordinary contract there is, the set came out empty. Empty is worse than
useless: `resolve_non_contract_references` then treats every `${VAR}` as
unmanaged and restores its literal value, undoing the deploy-time
parameterization and writing the author's passwords back into the baked
compose — while the bake prints "Sanitized". Fixed twice over: the set is
now passed in rather than re-derived, and the function reads both shapes.
**An unparseable contract silently became an empty one.** `bake.rs` did
`from_value(...).ok().unwrap_or_default()`, while the gate that is meant
to catch this was evaluated against the raw JSON — so the gate passed and
finalize sanitized against nothing. Reachable: every contract type denies
unknown fields, so a template using a newer kind fails wholesale on an
older binary. It now aborts.
**The keep-list matched more than it claimed.** The comment promised that
keeping `ollama` would not also keep `not-ollama-backup`; the generated
pattern `*[-_]ollama[-_]*` matched exactly that. The test asserted only
that the literal `*ollama*` was absent, so it passed against the bug.
Names now come from the contract and name compose volumes directly, so
the fuzzy patterns are gone — exact match, and the test checks the name
it names.
Also: two services declaring one volume with different policies is
refused rather than silently resolved toward `fixed`; `display` no longer
vanishes when a field collapses into the legacy `secret:` shorthand,
which was dropping the password-input hint on every `stacker sync`; and
`VolumePolicy`'s deserializer matches `Mutability` exhaustively, so a
fifth variant is a compile error instead of a runtime panic on user input.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A run takes 47 minutes, and the release binaries are compiled twice in it: once by the test job, then again from scratch inside the Docker build. The first set was already being packed into `app.tar.gz` and uploaded — nothing ever downloaded it. The reason it could not simply be reused is the one that has to be got right: the binaries are dynamically linked against glibc, and glibc is forward- but not backward-compatible. Built on the runner (Ubuntu 24.04, glibc 2.39) they would not start on debian:bookworm-slim (glibc 2.36). So the test job now runs inside `rust:bookworm` — the same image the Dockerfile builds in, and the one the runtime stage is derived from. The environments match exactly, and the Docker job copies the artifact instead of recompiling. The Dockerfile keeps both paths. `BINARIES=prebuilt` takes them from a build context; the default still compiles from source, so a local `docker build` works unchanged. Both were verified to parse, and the prebuilt path was built end to end: the four binaries, the config files and the sqlx CLI all land in the image with no compilation. Two things fell out along the way. The job now builds all four binaries the image needs — `console` and `backfill_field_policy` were missing from the artifact, which is part of why it could not be used. And `cargo install sqlx-cli` moved to its own small stage with only the postgres and rustls features, so the prebuilt path no longer pays 110 seconds of it to fetch two YAML files. Inside a job container, service containers resolve by name rather than 127.0.0.1, so PGHOST changes accordingly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fix/serialize volume policy
The reference documented `required_env:` under `config_contract`. No such field exists — the parser knows `required`, and since every contract type denies unknown fields, anyone following the example would have had their submission rejected. Removing it was right. It took the whole section with it, though, leaving the reference with no description of field policy at all: only the volume subsection added last week remained. An author reading this file would not learn that `mutability` exists. The design document in config/docs covers it, but that is not where someone writing a stacker.yml looks. Restored with the four mutabilities, the keys that apply to each, the legacy three-list shorthand, and a note that publishing is refused until secret-shaped fields carry a policy. Every example here was run through the parser, including the removed `required_env`, which is confirmed to be rejected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The run failed on `sudo: not found`. The container runs as root and does not ship sudo, and it does not need to: `rust:bookworm` already has pkg-config, libssl-dev and a C toolchain, and protoc never comes from the system — `build.rs` points PROTOC at a vendored binary unless one is set. Verified against the image. The step is gone. The error that was actually reported was `no such command: nextest`, which is not what went wrong. Both test steps carried `if: always()`, so they ran after the setup step failed and the nextest install had been skipped. Dropped on the first, narrowed to `success() || failure()` on the second, which is what was wanted: run both suites even if one fails, without reporting on an environment that was never built. Two more, found while looking rather than by the next 47-minute run: `--features explain` applies to the whole `cargo build`, not to the `--bin` it follows, so folding four binaries into one invocation was quietly shipping `server` with explain-logging on — a different binary from the one the image has always carried. Split in two; only `console` and the re-featured casbin dependency recompile. `.dockerignore` is empty, so the unpacked binaries and `app.tar.gz` were being sent to buildkit as part of `context: .` — hundreds of megabytes, twice, eating back the time this change exists to save. They now unpack to `runner.temp`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ci/build once reuse binaries
`"my-service": {}` is meaningless on its own — a service with no policy
simply goes unmentioned. In practice it means a declaration was lost
between the file and the registry.
That is not hypothetical. An older CLI serialized the contract without
`volumes:`, so a submit succeeded, the stored contract held an empty
block where the declaration should have been, and the bake reset a volume
the author had asked to keep. Nothing surfaced: not the submit, not the
bake, not the clone — until someone queried the database and noticed the
`{}`.
It happened twice, on two different stacks, for the same reason. The
second time cost another resubmit to discover.
Parsing now rejects it and names the service. The check sits in
`TargetConfigContract`'s deserializer, so it covers the CLI reading a
stacker.yml and the server validating a submitted contract alike.
All four marketplace templates still parse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An `env_file` line `commonDomain=` resolves to the empty string rather
than failing, so `domain: "${commonDomain}"` became `domain: ""` and
travelled all the way to the target. There the NPM role posted a proxy
host with no name, NPM rejected it, and because the task is `no_log` the
failure surfaced as a censored error that took the whole deploy down —
after the server had been provisioned (install 4068).
Two places let the blank through, so both are closed:
- `validate_semantics` gains E008: when a proxy is enabled, every
`proxy.domains` entry must carry a domain and a usable upstream. Deploy
already refuses to run on a blocking issue, so this now fails before
any server exists. An empty `domains:` list stays legal — a proxy can
be deployed and configured through its own UI later.
- `build_deploy_form` drops blank domains from `proxy_domains`, and omits
the key entirely when nothing is left to route. A route with no name is
not a route and must never leave the CLI.
The reference's validation table was three codes behind; E005 through
E007 are written down alongside the new E008.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fix/reject empty service block
Sanitizing removes the operator's own SSH key — deliberately, since a key left in the image would grant its holder root on every clone. The provider lookup happened afterwards, so a target Hetzner could not match left a box that was already cleaned, no longer reachable, and never snapshotted. One wrong flag reaches it: `--server-id` takes the *provider's* server id, and Stacker's own server id is a different number entirely. Passing 702 sanitized a build box and then failed with "server not found", costing a redeploy and a 5 GB model pull. `resolve_snapshot_target` exposes the lookup the snapshot call already performed internally, and bake now runs it first — before the box is touched — reporting the resolved id so an operator can see which machine is about to be captured. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(bake): confirm the box exists before sanitizing it
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 37456823 | Triggered | Generic Password | 27017fa | src/helpers/bake_finalize.rs | View secret |
| 37449298 | Triggered | Generic Password | 4b1e286 | src/cli/generator/compose.rs | View secret |
| 37449299 | Triggered | Generic High Entropy Secret | 4b1e286 | src/cli/generator/compose.rs | View secret |
| 37456824 | Triggered | Generic High Entropy Secret | 13a65d3 | src/cli/generator/compose.rs | View secret |
| 37490585 | Triggered | Generic High Entropy Secret | 51d5ff1 | src/cli/generator/compose.rs | View secret |
| 37490586 | Triggered | Generic Password | 51d5ff1 | src/cli/generator/compose.rs | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces significant improvements to the build and deployment process, especially around Docker image creation and CI efficiency. It also adds a new pre-commit hook for secret detection, updates the project version, and documents a major new release in the changelog. The Dockerfile and GitHub Actions workflow have been refactored to avoid redundant compilation, speed up CI, and ensure binary compatibility between build and runtime environments.
Build and Deployment Pipeline Improvements:
prebuilt(from CI) andbuilder(local/source build), improving build speed and reliability. (Dockerfile,.github/workflows/docker.yml) [1] [2] [3] [4] [5] [6] [7]rust:bookwormcontainer, ensuring glibc compatibility between built binaries and the runtime image. (.github/workflows/docker.yml)targetdirectory), preventing disk space exhaustion and speeding up builds. It usesSwatinem/rust-cachefor more reliable caching. (.github/workflows/docker.yml) [1] [2].github/workflows/docker.yml) [1] [2].github/workflows/docker.yml)Testing and Workflow Reliability:
.github/workflows/docker.yml)Security and Tooling:
.pre-commit-config.yaml)Documentation and Versioning:
CHANGELOG.md) [1] [2]Cargo.toml)docs/ONE_CLICK_DEPLOY.md)These changes result in faster, more reliable CI/CD, improved security, and clearer documentation for the new release.