Skip to content

CI: Fail on missing preconditions and exceeded boundaries - #1754

Open
Steven Shriver (stshrive) wants to merge 2 commits into
agent-substrate:mainfrom
stshrive:dev/harden-preconditions
Open

Steven Shriver (stshrive) wants to merge 2 commits into
agent-substrate:mainfrom
stshrive:dev/harden-preconditions

Conversation

@stshrive

@stshrive Steven Shriver (stshrive) commented Sep 18, 2026

Copy link
Copy Markdown

This change addresses six ways CI pipeline could report false success or hang on broken infrastructure. Importantly, this change does two things to reduce load on infrastructure:

  1. It prevents jobs from running for the default action limit of 360m by pinning the timeout to 45m.
  2. It limits pull_request jobs from continuing execution once superseded by new changes.
  • Silent container test skipping: Tests now explicitly fail if CI or REQUIRE_DOCKER is set (while still skipping on local machines lacking Docker), with the check implemented in dockerenv to avoid an import cycle between storetest and atepg.
  • Unbounded trust bundle wait: Enforced a shared 120-second timeout across both bundles (overridable via ATE_INSTALL_TRUST_BUNDLE_TIMEOUT) and added diagnostic dumping of bundles, controller pods, and logs before returning a non-zero exit code.
  • Missing sandbox preflight validation: Added early preflight checks for /dev/kvm and SandboxConfig/microvm that fail fast and print actionable remediation instructions.
  • Skipped migration checks on main: Configured the migration immutability check to run on pushes to main to catch modified migrations at the point of merge.
  • Missing job timeouts and concurrency limits: Defined explicit timeout-minutes (45m and 120m bounds) and added concurrency groups that automatically cancel superseded pull request runs without canceling runs on main.

Fixes #1747

  • Tests pass
    • Silent container skipping, unbounded trust bundles, and missing sandbox were all forced locally and confirmed to exist with changes here resolving each.
    • The latter half of the scenarios exist in CI only due to being GH Action trigger issues.

@BenTheElder

Copy link
Copy Markdown
Collaborator

Post-merge vulnerability scanning: Configured govulncheck to run on pull requests to catch vulnerable dependencies prior to merging into main.

We intentionally did not do this. Vulns are point-in-time X sources, not just sources.
We don't want to stop merging other PRs when there's a vuln in main's deps.

@stshrive

Steven Shriver (stshrive) commented Sep 21, 2026

Copy link
Copy Markdown
Author

We intentionally did not do this. Vulns are point-in-time X sources, not just sources.
We don't want to stop merging other PRs when there's a vuln in main's deps.

There are two vectors of getting a vuln though.

The first is a new dependency that is not in main and has not been scanned yet gets added in a PR without being scanned.

The second is a new point-in-time vuln being discovered in a dependency already in main.

I think it is worth manually overriding the latter to prevent the former from being inadvertently added to main.

@stshrive

Copy link
Copy Markdown
Author

I removed the update to the vuln check execution pattern. This should remedy any concerns about new point-in-time vulns in main blocking PRs.

@stshrive
Steven Shriver (stshrive) marked this pull request as ready for review September 21, 2026 19:41
Comment thread cmd/ateapi/internal/store/dockerenv/dockerenv.go
Comment thread internal/e2e/preflight.go Outdated
Comment thread internal/e2e/preflight.go Outdated
Six ways the pipeline reported false success or hung on broken
infrastructure.

- **Silent container test skipping**: Tests now explicitly fail if
  `CI` or `REQUIRE_DOCKER` is set (while still skipping on local
  machines lacking Docker), with the check implemented in
  `dockerenv` to avoid an import cycle between `storetest` and
  `atepg`.
- **Unbounded trust bundle wait**: Enforced a shared 120-second
  timeout across both bundles (overridable via
  `ATE_INSTALL_TRUST_BUNDLE_TIMEOUT`) and added diagnostic dumping
  of bundles, controller pods, and logs before returning a non-zero
  exit code.
- **Missing sandbox preflight validation**: Added early preflight
  checks for `SandboxConfig/microvm` that fail fast for missing
  configs. Missing device diagnostics are deferred to runtime as
  not to pin to a particular implementation.
- **Skipped migration checks on main**: Configured the migration
  immutability check to run on pushes to main to catch modified
  migrations at the point of merge.
- **Missing job timeouts and concurrency limits**: Defined explicit
  timeout-minutes (45m and 120m bounds) and added concurrency
  groups that automatically cancel superseded pull request runs
  without canceling runs on main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Testing preconditions are not a hard-stop for some test causing low-fidelity signals in the CI/testing infra.

3 participants