Bake improvement

CHANGELOG.md

@@ -4,7 +4,35 @@ All notable changes to this project will be documented in this file.
 
 ## [Unreleased]
 
-### Added — Developer project synchronization
+## [0.3.3] — 2026-09-18
+
+### Added — Chat session management
+
+- Added chat session endpoints with archive and per-message encryption.
+- Added Casbin RBAC rules for `/api/chat` routes.
+
+### Added — Agent hardening & ownership
+
+- Enforced per-tenant ownership on `/v1/agent` routes.
+- Agent tokens now verified against a stored digest instead of round-tripping Vault.
+- Agent authentication fails closed on Vault errors.
+- Agent registration requires the service key; marketplace registration returns 501.
+- Accept aggregate `all_health` report from agents for container status.
+- Return `project_id` in agent snapshots and one-click clone endpoint.
+- Container scope classified from Docker label, not container name.
+- Moved `rotate-token` from the console binary to the CLI (`stacker agent rotate-token`).
+
+### Added — Marketplace field policy & secret federation
+
+- Added `config_contract` field-policy support (`fixed`/`editable`/`generated` + types: `hex`, `alphanumeric`, `uuid`, `derived_jwt`).
+- At publish, `generated`-field values are stripped from the stored `stack_definition` (fail-closed).
+- `derived_jwt` fields signed on cloned boxes via HMAC.
+- Config contract federated to the User Service.
+- Added `backfill_field_policy` one-shot tool to re-gate the existing catalog.
+- Policy-driven `generate-secrets.sh` reads `mutability:generated` field policy instead of hardcoding `openssl rand`.
+- Added `DisplayType` enum and `display` field to `FieldPolicy`.
+
+### Added — Project synchronization & one-click deploy
 
 - Added `stacker sync` to synchronize declarative project and app configuration
   with Stacker without creating a deployment, contacting a target server, or
@@ -14,6 +42,57 @@ All notable changes to this project will be documented in this file.
   synchronization.
 - Added centralized sensitive environment-name redaction and validation for
   marketplace asset and seed-job metadata.
+- One-click clone now registers the server in inventory and creates a cloud firewall.
+- One-click clone seeds `project_app` records for the Applications panel.
+
+### Added — Deployment lifecycle & cleanup
+
+- Added `deployment_container` table to track containers per deployment (replaces name inference).
+- Added deployment container sweeper for stale containers.
+- Added stale project and server cleanup with notification (`cleanup-notify` binary).
+- Added scheduled audit-log cleanup cron job.
+- Added env size validator.
+
+### Added — Security & infrastructure
+
+- mTLS for Vault access; Vault client reports missing CA.
+- `yaml_quote` now escapes control characters (`\n`, `\r`, `\t`).
+- Docker preflight check (`docker info`) before deploy.
+- New `W003` warning: `deploy.server.ssh_key` silently ignored on cloud deploys.
+
+### Fixed — SSH key authorization
+
+- Cloud deploy now fails when SSH key cannot be stored (was a silent warning).
+- Cloud deploy fails when no SSH access is verified after provisioning.
+- SSH key authorization retried while the VM boots; both Vault-managed and user keys authorized independently.
+- User's configured SSH key from `deploy.cloud.ssh_key` authorized through the correct endpoint.
+
+### Fixed — CLI & config
+
+- Fixed `server --dry-run` no longer runs a real Docker deploy (#238).
+- Fixed 500 on `PUT /cloud/{id}`: owner set before conversion.
+- Fixed #251: escape env/label values so multiline config survives YAML.
+- Ports validated by range, not by digit count.
+- Port values in `stacker.yml` handled correctly when unquoted.
+- Healthcheck `test` field emitted in the form docker compose expects (CMD list vs CMD-SHELL).
+
+### Fixed — Database & migrations
+
+- Fixed migration version collisions breaking CI.
+- Guard optional cron job lookup to prevent panics.
+- Reconcile audit-log cleanup cron after extension install.
+- Restored `sqlx prepare` with missing `config_contract` field.
+
+### Fixed — Auth & Casbin
+
+- Added missing Casbin rules for admin `detect-secrets` endpoint.
+- Agent auth accepts both Vault response shapes for the token.
+- Credentials tests no longer read the developer's own config.
+
+### Fixed — Marketplace BDD
+
+- BDD marketplace analytics fixtures: cast `template_id` to UUID.
+- BDD marketplace scenarios: seed `source_project_id` + deployment.
 
 ## [0.3.2] — 2026-08-26
 

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "stacker"
-version = "0.3.2"
+version = "0.3.3"
 edition = "2021"
 default-run= "server"
 

migrations/20260917120000_fix_cleanup_type_mismatch_cleanup_log_retention.down.sql

@@ -0,0 +1,78 @@
+-- Revert: restore original stacker_cleanup_terminal_commands, drop cleanup_log cleanup
+
+-- Unschedule cleanup_log cron job
+DO $$
+BEGIN
+    IF EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pg_cron') THEN
+        PERFORM cron.unschedule('stacker_cleanup_cleanup_log');
+    END IF;
+EXCEPTION WHEN OTHERS THEN NULL;
+END;
+$$;
+
+-- Drop cleanup_log cleanup function
+DROP FUNCTION IF EXISTS stacker_cleanup_cleanup_log(INTERVAL, BOOLEAN);
+
+-- Restore original stacker_cleanup_terminal_commands (with the type mismatch bug)
+CREATE OR REPLACE FUNCTION stacker_cleanup_terminal_commands(
+    retention INTERVAL DEFAULT INTERVAL '30 days',
+    p_dry_run BOOLEAN DEFAULT true
+)
+RETURNS void LANGUAGE plpgsql AS $$
+DECLARE
+    v_count BIGINT;
+BEGIN
+    IF p_dry_run THEN
+        SELECT count(*) INTO v_count FROM commands
+        WHERE status IN ('completed','failed','cancelled')
+          AND COALESCE(completed_at, updated_at) < NOW() - retention;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'commands', v_count, retention, true);
+
+        SELECT count(*) INTO v_count FROM dead_letter_queue
+        WHERE status IN ('exhausted','discarded','resolved')
+          AND updated_at < NOW() - retention;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'dead_letter_queue', v_count, retention, true);
+
+        SELECT count(*) INTO v_count FROM pipe_executions
+        WHERE completed_at < NOW() - retention;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'pipe_executions', v_count, retention, true);
+
+        SELECT count(*) INTO v_count FROM pipe_dag_step_executions
+        WHERE completed_at < NOW() - retention;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'pipe_dag_step_executions', v_count, retention, true);
+    ELSE
+        -- Delete command_queue entries for terminal commands first (FK)
+        DELETE FROM command_queue WHERE command_id IN (
+            SELECT id FROM commands
+            WHERE status IN ('completed','failed','cancelled')
+              AND COALESCE(completed_at, updated_at) < NOW() - retention
+        );
+
+        DELETE FROM commands
+        WHERE status IN ('completed','failed','cancelled')
+          AND COALESCE(completed_at, updated_at) < NOW() - retention;
+        GET DIAGNOSTICS v_count = ROW_COUNT;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'commands', v_count, retention, false);
+
+        DELETE FROM dead_letter_queue
+        WHERE status IN ('exhausted','discarded','resolved')
+          AND updated_at < NOW() - retention;
+        GET DIAGNOSTICS v_count = ROW_COUNT;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'dead_letter_queue', v_count, retention, false);
+
+        -- FK-safe order: step executions before executions
+        DELETE FROM pipe_dag_step_executions WHERE completed_at < NOW() - retention;
+
+        DELETE FROM pipe_executions WHERE completed_at < NOW() - retention;
+        GET DIAGNOSTICS v_count = ROW_COUNT;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'pipe_executions', v_count, retention, false);
+    END IF;
+END;
+$$;

migrations/20260917120000_fix_cleanup_type_mismatch_cleanup_log_retention.up.sql

@@ -0,0 +1,106 @@
+-- Fix stacker_cleanup_terminal_commands type mismatch and add cleanup_log retention
+
+-- 1. Fix stacker_cleanup_terminal_commands: command_queue.command_id is VARCHAR
+--    referencing commands.command_id (VARCHAR), NOT commands.id (UUID).
+--    The original function used `SELECT id FROM commands` which caused:
+--    ERROR: operator does not exist: character varying = uuid
+CREATE OR REPLACE FUNCTION stacker_cleanup_terminal_commands(
+    retention INTERVAL DEFAULT INTERVAL '30 days',
+    p_dry_run BOOLEAN DEFAULT true
+)
+RETURNS void LANGUAGE plpgsql AS $$
+DECLARE
+    v_count BIGINT;
+BEGIN
+    IF p_dry_run THEN
+        SELECT count(*) INTO v_count FROM commands
+        WHERE status IN ('completed','failed','cancelled')
+          AND COALESCE(completed_at, updated_at) < NOW() - retention;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'commands', v_count, retention, true);
+
+        SELECT count(*) INTO v_count FROM dead_letter_queue
+        WHERE status IN ('exhausted','discarded','resolved')
+          AND updated_at < NOW() - retention;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'dead_letter_queue', v_count, retention, true);
+
+        SELECT count(*) INTO v_count FROM pipe_executions
+        WHERE completed_at < NOW() - retention;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'pipe_executions', v_count, retention, true);
+
+        SELECT count(*) INTO v_count FROM pipe_dag_step_executions
+        WHERE completed_at < NOW() - retention;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'pipe_dag_step_executions', v_count, retention, true);
+    ELSE
+        -- Delete command_queue entries for terminal commands first (FK)
+        -- Use command_id (VARCHAR) not id (UUID) to match command_queue.command_id type
+        DELETE FROM command_queue WHERE command_id IN (
+            SELECT command_id FROM commands
+            WHERE status IN ('completed','failed','cancelled')
+              AND COALESCE(completed_at, updated_at) < NOW() - retention
+        );
+
+        DELETE FROM commands
+        WHERE status IN ('completed','failed','cancelled')
+          AND COALESCE(completed_at, updated_at) < NOW() - retention;
+        GET DIAGNOSTICS v_count = ROW_COUNT;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'commands', v_count, retention, false);
+
+        DELETE FROM dead_letter_queue
+        WHERE status IN ('exhausted','discarded','resolved')
+          AND updated_at < NOW() - retention;
+        GET DIAGNOSTICS v_count = ROW_COUNT;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'dead_letter_queue', v_count, retention, false);
+
+        -- FK-safe order: step executions before executions
+        DELETE FROM pipe_dag_step_executions WHERE completed_at < NOW() - retention;
+
+        DELETE FROM pipe_executions WHERE completed_at < NOW() - retention;
+        GET DIAGNOSTICS v_count = ROW_COUNT;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_terminal_commands', 'pipe_executions', v_count, retention, false);
+    END IF;
+END;
+$$;
+
+
+-- 2. Add cleanup_log self-cleanup (90 days retention)
+--    cleanup_log accumulates entries from all 7 cleanup functions daily but has no purge.
+CREATE OR REPLACE FUNCTION stacker_cleanup_cleanup_log(
+    retention INTERVAL DEFAULT INTERVAL '90 days',
+    p_dry_run BOOLEAN DEFAULT true
+)
+RETURNS void LANGUAGE plpgsql AS $$
+DECLARE
+    v_count BIGINT;
+BEGIN
+    IF p_dry_run THEN
+        SELECT count(*) INTO v_count FROM cleanup_log WHERE run_at < NOW() - retention;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_cleanup_log', 'cleanup_log', v_count, retention, true);
+    ELSE
+        DELETE FROM cleanup_log WHERE run_at < NOW() - retention;
+        GET DIAGNOSTICS v_count = ROW_COUNT;
+        INSERT INTO cleanup_log (function_name, table_name, rows_deleted, retention, dry_run)
+        VALUES ('stacker_cleanup_cleanup_log', 'cleanup_log', v_count, retention, false);
+    END IF;
+END;
+$$;
+
+
+-- 3. Schedule cleanup_log purge (weekly Sunday 5:30 AM)
+DO $$
+BEGIN
+    IF EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pg_cron') THEN
+        IF NOT EXISTS (SELECT 1 FROM cron.job WHERE jobname = 'stacker_cleanup_cleanup_log') THEN
+            PERFORM cron.schedule('stacker_cleanup_cleanup_log', '30 5 * * 0',
+                $cron$SELECT stacker_cleanup_cleanup_log(p_dry_run := false);$cron$);
+        END IF;
+    END IF;
+END;
+$$;

migrations/20260919120000_baked_snapshots_required_env_keys.down.sql

@@ -0,0 +1 @@
+ALTER TABLE baked_snapshots DROP COLUMN required_env_keys;

migrations/20260919120000_baked_snapshots_required_env_keys.up.sql

@@ -0,0 +1,10 @@
+-- Record which ${VAR} references the baked compose actually needs, so the clone
+-- path can refuse a deploy whose env file would not satisfy them.
+--
+-- Without this, a missing key is silent: Docker Compose substitutes an empty
+-- string with only a warning, the unit's ExecStartPre ends in `|| true`, and the
+-- systemd unit still reports active while the stack is misconfigured.
+--
+-- Nullable on purpose: snapshots baked before this column carry NULL and skip
+-- the check, so existing images keep deploying unchanged.
+ALTER TABLE baked_snapshots ADD COLUMN required_env_keys JSONB;