Skip to content

test(ci): backfill negative-path tests for happy-path-only actionsΒ #183

@devantler

Description

@devantler

πŸ€– Generated by the Daily AI Assistant

Part of #181.

Problem

.github/workflows/ci.yaml shows uneven negative-path test coverage across the 15 composite actions:

Action Negative-path test? Notes
aggregate-job-checks βœ… test-…-failure, test-…-unknown (uses continue-on-error + outcome assertion)
setup-agent-skills βœ… test-…-missing-input, test-…-malformed-line
update-agent-skills βœ… test-…-missing-dir
approve-pr ❌ only the happy-path
create-issues-from-todos ❌ only the happy-path
enable-auto-merge-on-pr ❌ only the happy-path (dry-run)
login-to-ghcr ❌ only the happy-path
run-dotnet-tests ❌ only the matrix happy-path
setup-go-toolchain ❌ happy-path + private-modules variant; no failure mode
setup-ksail-cli ❌ matrix happy-path
sync-github-labels ❌ only the happy-path
upload-coverage ❌ smoke test only
upsert-issue βœ… (sort of) create + close β€” covers two states, not error paths

The well-covered ones use a consistent pattern (continue-on-error: true + outcome assertion) that's directly transferable. Regressions on error handling (bad inputs, missing files, missing config, malformed YAML, wrong arguments) wouldn't be caught today.

Proposed direction

For each action in the ❌ rows above, add one negative-path test that exercises a clear, mechanically-reachable failure mode. Examples:

  • sync-github-labels β€” missing/invalid config-file.
  • login-to-ghcr β€” missing/empty github-token (verify clean failure, not a crash).
  • enable-auto-merge-on-pr β€” invalid merge-method (e.g. merge-method: "rebasee").
  • upload-coverage β€” missing file:; or unknown language:.
  • setup-go-toolchain β€” invalid go-version: string (e.g. "banana").
  • upsert-issue β€” missing both body and body-file (the action's documented error path).
  • setup-ksail-cli β€” no obvious negative path; may be skippable, document why in the PR body if so.
  • run-dotnet-tests β€” non-existent working-directory.
  • approve-pr / create-issues-from-todos β€” gated by if: on devantler-only PRs and use real GitHub App auth, harder to write a public negative test without leaking PAT scope; investigate and document in the PR if not feasible.

Each test follows the existing pattern (use continue-on-error: true on the action invocation, then a follow-up bash step that asserts steps.<id>.outcome == 'failure'). One PR per action or one consolidated PR β€” the PR author's judgement; consolidated is preferred for review efficiency unless the test-count gets unwieldy.

Acceptance criteria

  • At least one negative-path test added for: sync-github-labels, login-to-ghcr, enable-auto-merge-on-pr, upload-coverage, setup-go-toolchain, upsert-issue, run-dotnet-tests.
  • Actions for which a clean negative-path test is not feasible (e.g. setup-ksail-cli, approve-pr, create-issues-from-todos) have a one-paragraph justification in the PR body for why they were skipped.
  • Each new test uses the established pattern (continue-on-error: true β†’ outcome assertion).
  • CI passes; zizmor passes.

Out of scope

  • Extracting a shared "negative-path harness" reusable workflow β€” if scaffolding duplicates >2–3 times, capture as a follow-up against devantler-tech/reusable-workflows per the monorepo AGENTS.md Holistic review rule, but don't pre-emptively extract.
  • Re-running actionlint policy in CI (already part of zizmor / standard checks).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    πŸ‘€ In Review

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions