What happened
In PR #3324, CodeRabbit identified that the new TA task acceptance test used the same POLICY_BUNDLE_DIGEST value (sha256:f904979d...) as the task's default at tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml:181. This makes the test tautological — it would pass even if parameter propagation was broken, because the task would fall through to its default value. simonbaird fixed the TA test in commit 5d85d39 by changing to a distinct digest. However, the existing VEC test at features/task_validate_image.feature:95 still uses sha256:f904979d..., which is the same default value defined in tasks/verify-enterprise-contract/0.1/verify-enterprise-contract.yaml:181. The original PR author was correctly mirroring this existing VEC pattern, which itself has the same tautological issue.
What could go better
The VEC test should use a distinct digest value, just as the TA test was fixed to do. Additionally, the repo's acceptance test guidance (.claude/rules/acceptance-tests.md and acceptance/README.md) contains no guidance about ensuring test parameter values differ from production defaults when the purpose of the test is to verify parameter propagation. Both the code agent (which co-authored the original commit) and the review agent missed this because they were correctly following the existing (flawed) pattern. I am confident in this finding — the VEC task default at line 181 is identical to the VEC test value at line 95, confirmed by direct file reads.
Proposed change
- Update
features/task_validate_image.feature:95 to use a distinct POLICY_BUNDLE_DIGEST value (e.g., sha256:2a851b5d56259ba46391f72bc4e75ed5475510dce5ae47d054b72baf1e908d26 or another non-default digest) and update the corresponding snapshot in features/__snapshots__/task_validate_image.snap.
- Add a guideline to
.claude/rules/acceptance-tests.md or acceptance/README.md: when writing acceptance tests that supply a parameter to a task, use a value distinct from the task's default to verify that the parameter is actually propagated (not just falling through to the default).
Validation criteria
- After the fix,
grep -c 'f904979d' features/task_validate_image.feature returns 0 (no test uses the task default digest). 2. The acceptance tests pass with make scenario_Pin_policy_bundle_digest for the VEC task variant. 3. Future PRs adding parameterized acceptance tests for tasks with defaults should use distinct values, verifiable by reviewing the next 3 such PRs.
Generated by retro agent from #3324
What happened
In PR #3324, CodeRabbit identified that the new TA task acceptance test used the same
POLICY_BUNDLE_DIGESTvalue (sha256:f904979d...) as the task's default attasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml:181. This makes the test tautological — it would pass even if parameter propagation was broken, because the task would fall through to its default value. simonbaird fixed the TA test in commit5d85d39by changing to a distinct digest. However, the existing VEC test atfeatures/task_validate_image.feature:95still usessha256:f904979d..., which is the same default value defined intasks/verify-enterprise-contract/0.1/verify-enterprise-contract.yaml:181. The original PR author was correctly mirroring this existing VEC pattern, which itself has the same tautological issue.What could go better
The VEC test should use a distinct digest value, just as the TA test was fixed to do. Additionally, the repo's acceptance test guidance (
.claude/rules/acceptance-tests.mdandacceptance/README.md) contains no guidance about ensuring test parameter values differ from production defaults when the purpose of the test is to verify parameter propagation. Both the code agent (which co-authored the original commit) and the review agent missed this because they were correctly following the existing (flawed) pattern. I am confident in this finding — the VEC task default at line 181 is identical to the VEC test value at line 95, confirmed by direct file reads.Proposed change
features/task_validate_image.feature:95to use a distinctPOLICY_BUNDLE_DIGESTvalue (e.g.,sha256:2a851b5d56259ba46391f72bc4e75ed5475510dce5ae47d054b72baf1e908d26or another non-default digest) and update the corresponding snapshot infeatures/__snapshots__/task_validate_image.snap..claude/rules/acceptance-tests.mdoracceptance/README.md: when writing acceptance tests that supply a parameter to a task, use a value distinct from the task's default to verify that the parameter is actually propagated (not just falling through to the default).Validation criteria
grep -c 'f904979d' features/task_validate_image.featurereturns 0 (no test uses the task default digest). 2. The acceptance tests pass withmake scenario_Pin_policy_bundle_digestfor the VEC task variant. 3. Future PRs adding parameterized acceptance tests for tasks with defaults should use distinct values, verifiable by reviewing the next 3 such PRs.Generated by retro agent from #3324