Skip to content

Fix Prefect flaky test#23219

Open
lucia-sb wants to merge 4 commits intomasterfrom
lucia-sb/fix-prefect
Open

Fix Prefect flaky test#23219
lucia-sb wants to merge 4 commits intomasterfrom
lucia-sb/fix-prefect

Conversation

@lucia-sb
Copy link
Copy Markdown
Contributor

@lucia-sb lucia-sb commented Apr 8, 2026

What does this PR do?

Adds an extra readiness check to the Prefect E2E test environment so the test waits until task runs and task-run events are actually queryable from the Prefect API before executing metric assertions.

Motivation

The Prefect E2E test is flaky because flow execution logs can appear before Prefect has fully persisted task-run records and task-run events. That race causes task-run metrics to be missing even though the flows already finished.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

@lucia-sb lucia-sb requested a review from a team as a code owner April 8, 2026 08:38
@lucia-sb lucia-sb added the qa/skip-qa Automatically skip this PR for the next QA label Apr 8, 2026
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 09bcc22574

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@datadog-official
Copy link
Copy Markdown
Contributor

datadog-official bot commented Apr 8, 2026

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 87.09%

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 9790ba6 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 8, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.02%. Comparing base (32a22f7) to head (9790ba6).
⚠️ Report is 21 commits behind head on master.

Additional details and impacted files
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

COMPOSE_FILE_E2E, patterns=["Finished all tasks"], service="prefect-worker", attempts=120, wait=1
),
CheckDockerLogs(COMPOSE_FILE_E2E, patterns=["Retried"], service="prefect-worker", attempts=120, wait=1),
WaitFor(lambda: _check_task_runs_available(prefect_url), attempts=60, wait=2),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

60 attempts with a timeout of 10 seconds for each seems a bit long
How often do we expect the startup to take close to 10 minutes?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You’re right, it shouldn’t take that long. I’ve reduced the timeout to 1 second, since the Prefect server is already healthy when the call is made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

integration/prefect qa/skip-qa Automatically skip this PR for the next QA team/agent-integrations

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants