Skip to content

Turn on Helix Job Monitor - #38751

Open
wtgodbe wants to merge 4 commits into
mainfrom
feature/helix-job-monitor
Open

wtgodbe wants to merge 4 commits into
mainfrom
feature/helix-job-monitor

Conversation

@wtgodbe

@wtgodbe wtgodbe commented Aug 5, 2026

Copy link
Copy Markdown
Member

Enable Arcade's Helix Job Monitor for the public and internal test pipelines. Helix submission jobs can now release their agents after queueing work, while the monitor publishes test results and owns the final Helix status.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@wtgodbe
wtgodbe marked this pull request as ready for review August 5, 2026 17:25
@wtgodbe
wtgodbe requested a review from AndriySvyryd as a code owner August 5, 2026 17:25
Copilot AI lite review requested due to automatic review settings August 5, 2026 17:25
@wtgodbe

wtgodbe commented Aug 5, 2026

Copy link
Copy Markdown
Member Author

@AndriySvyryd PTAL

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Enables Arcade’s Helix Job Monitor in EF Core’s public and internal Azure DevOps pipelines so Helix submission jobs can stop after queueing, while a dedicated monitor job publishes test results and drives the final Helix status.

Changes:

  • Add Microsoft.DotNet.Helix.JobMonitor dependency/version plumbing and pin the tool via .config/dotnet-tools.json.
  • Enable Helix Job Monitor behavior in eng/helix.proj when SYSTEM_ACCESSTOKEN is available.
  • Add the helix-job-monitor.yml job template to both public and internal pipelines.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
eng/Version.Details.xml Adds Helix Job Monitor dependency tracking entry.
eng/Version.Details.props Introduces Helix Job Monitor version properties alongside other dotnet-dotnet dependencies.
eng/helix.proj Turns on Helix Job Monitor mode when SYSTEM_ACCESSTOKEN is set.
azure-pipelines-public.yml Adds Helix Job Monitor job template to the public pipeline (currently with an indentation issue).
azure-pipelines-internal-tests.yml Adds Helix Job Monitor job template to the internal test pipeline and passes helixAccessToken.
.config/dotnet-tools.json Pins the dotnet-helix-job-monitor tool version for dotnet tool restore.

Comment thread azure-pipelines-public.yml Outdated
- template: /eng/common/core-templates/job/helix-job-monitor.yml@self
parameters:
helixAccessToken: $(HelixApiAccessToken)
- stage: validate

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to change the validate logic to also check whether the corresponding Helix monitor job succeeded

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, we should check the result of each individual Helix job to preserve the validation logic, though I am not sure whether this is currently possible to do here (feature request?)
Otherwise, add $helixJobMonitorResult to each item in $groupResults instead of failing outright

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally, we should check the result of each individual Helix job to preserve the validation logic

What would be the benefit of this? The helix jobs no longer depend on the test results, they just send the tests off and then report green. If one fails, the monitor will fail too

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be the benefit of this? The helix jobs no longer depend on the test results, they just send the tests off and then report green. If one fails, the monitor will fail too

Right, the benefit would be from checking the helix monitoring jobs for test failures specific to that leg

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the benefit would be from checking the helix monitoring jobs for test failures specific to that leg

But the individual legs don't fail when there are test failures - as soon as the tests are sent to helix, they complete w/ success. Only the helix monitor job will ever fail for test failures.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only the helix monitor job will ever fail for test failures.

Yes and we need to make that failure more granular, so that we can continue to check only the relevant failures in the validation groups.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 5, 2026 17:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

@wtgodbe
wtgodbe marked this pull request as draft August 6, 2026 17:36
Preserve the Helix job monitor validation while retaining the retry aggregation improvements from main. Update the new JobMonitor dependency to the current dotnet build versions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@wtgodbe
wtgodbe marked this pull request as ready for review September 15, 2026 21:16
Copilot AI review requested due to automatic review settings September 15, 2026 21:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Monitor job dependencies and no-Helix-job handling must be corrected in both pipelines.

Get a fresh assessment by requesting another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Lite

Comment thread azure-pipelines-internal-tests.yml Outdated
HelixAccessToken: $(_HelixAccessToken)
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
DotNetBuildsInternalReadSasToken: $(dotnetbuilds-internal-container-read-token)
- template: /eng/common/core-templates/job/helix-job-monitor.yml@self
Comment thread azure-pipelines-internal-tests.yml Outdated
Comment thread azure-pipelines-public.yml Outdated
env:
HelixAccessToken: $(_HelixAccessToken)
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
- template: /eng/common/core-templates/job/helix-job-monitor.yml
Comment thread azure-pipelines-public.yml Outdated
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings September 15, 2026 22:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Both pipeline monitor jobs can run before Helix submissions; add explicit dependencies and use always().

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

azure-pipelines-internal-tests.yml:667

  • This invocation leaves the monitor's dependsOn at its [] default, so Azure DevOps can start HelixJobMonitor in parallel with the Helix_* submission jobs; list order does not create a dependency. It can therefore observe no queued work and finish before the submitters release their agents. Pass explicit dependencies on the Helix jobs and use always() so skipped jobs in filtered runs do not suppress the monitor.
    azure-pipelines-public.yml:531
  • This invocation leaves the monitor's dependsOn at its [] default, so Azure DevOps can start HelixJobMonitor in parallel with the Helix_* submission jobs; list order does not create a dependency. It can therefore observe no queued work and finish before the submitters release their agents. Pass explicit dependencies on the Helix jobs and use always() so skipped jobs in filtered runs do not suppress the monitor.
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants