Skip to content

test(feature-processor): Isolate pipeline names to fix flaky integ tests#6095

Merged
lucasjia-aws merged 1 commit into
aws:masterfrom
lucasjia-aws:test/mlops
Jul 23, 2026
Merged

test(feature-processor): Isolate pipeline names to fix flaky integ tests#6095
lucasjia-aws merged 1 commit into
aws:masterfrom
lucasjia-aws:test/mlops

Conversation

@lucasjia-aws

Copy link
Copy Markdown
Collaborator

The feature processor to_pipeline integ tests hardcoded fixed pipeline names (pipeline-name-01, pipeline-name-lf-01) and thus shared fixed S3 paths (s3://.../<pipeline_name>/function/payload.pkl). Combined with the asymmetric-signing scheme introduced in PR #5816 (each to_pipeline call generates a fresh ECDSA key pair, overwrites the signed payload, and pins the matching public key into the pipeline env), concurrent CI builds on the same account overwrite each other's payloads. A running execution then verifies a payload signed by a different build's key, producing DeserializationError: "Integrity check for the serialized function or data failed" and pipeline execution status Failed.

Generate unique pipeline names via unique_name_from_base so each test run and build uses an isolated S3 prefix, and restore cleanup_pipeline in the finally blocks to avoid resource leakage.

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

The feature processor to_pipeline integ tests hardcoded fixed pipeline
names (pipeline-name-01, pipeline-name-lf-01) and thus shared fixed S3
paths (s3://.../<pipeline_name>/function/payload.pkl). Combined with the
asymmetric-signing scheme introduced in PR aws#5816 (each to_pipeline call
generates a fresh ECDSA key pair, overwrites the signed payload, and
pins the matching public key into the pipeline env), concurrent CI
builds on the same account overwrite each other's payloads. A running
execution then verifies a payload signed by a different build's key,
producing DeserializationError: "Integrity check for the serialized
function or data failed" and pipeline execution status Failed.

Generate unique pipeline names via unique_name_from_base so each test
run and build uses an isolated S3 prefix, and restore cleanup_pipeline
in the finally blocks to avoid resource leakage.
@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown

Claude finished @lucasjia-aws's task in 1m 1s —— View job


PR Review: Isolate feature-processor pipeline names

  • Read the PR diff (/tmp/pr.diff)
  • Review context (call sites, cleanup_pipeline, unique_name_from_base)
  • Assess correctness, tests, and conventions

This is a well-targeted fix. The root-cause analysis in the description is convincing: hardcoded pipeline names (pipeline-name-01, pipeline-name-lf-01) collide on shared S3 payload paths, and with the per-call ECDSA signing from #5816 concurrent builds clobber each other's signed payloads → DeserializationError. Making each run use unique_name_from_base(...) gives every build an isolated S3 prefix, which addresses the flakiness directly.

Correctness

  • unique_name_from_base("pipeline-name-01") yields pipeline-name-01-<ts>-<hex> (≤ 63 chars, only [a-zA-Z0-9-]), which satisfies SageMaker pipeline name constraints. ✅
  • The generated pipeline_name local is threaded consistently through to_pipeline, execute, Pipeline(name=...), and the cleanup_pipeline calls in each test. ✅
  • Restoring cleanup_pipeline in the finally blocks prevents resource leakage. In the Lake Formation test, placing it outside the if car_data_fg is not None: guard is correct — the pipeline is created regardless of feature-group state, so it should always be cleaned up.
  • cleanup_pipeline already swallows ResourceNotFound, so it's safe to call even if to_pipeline never created the pipeline (e.g. early failure).

Minor observations (non-blocking, pre-existing)

  • In test_to_pipeline_and_execute / test_schedule_and_event_trigger, the finally block references feature_groups, which is unbound if create_feature_groups(...) raises — that would mask the original error with a NameError. Not introduced by this PR, but worth a follow-up if these tests ever fail during setup.

No new imports beyond unique_name_from_base, which resolves correctly to sagemaker.core.common_utils. The change is minimal, correct, and consistent with the surrounding test conventions.

Verdict: LGTM.

@lucasjia-aws
lucasjia-aws merged commit 670ed8c into aws:master Jul 23, 2026
14 of 22 checks passed
@github-actions github-actions Bot mentioned this pull request Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants