CI: pin UCX transports — retire the MANA-NIC MPI_Init flake (#461) - #529
Open
lmoresi wants to merge 1 commit into
Open
CI: pin UCX transports — retire the MANA-NIC MPI_Init flake (#461)#529lmoresi wants to merge 1 commit into
lmoresi wants to merge 1 commit into
Conversation
Some Azure runners advertise an InfiniBand verbs device (mana_0) that UCX cannot open; MPI_Init aborts inside the import smoke test before any underworld3 code runs. Four hits in a week across #466/#526/#527 (issues #461/#528), each fixed by re-rolling the runner lottery. A single-node CI runner needs only shared memory and TCP, so pin the transports at the workflow level - it covers the smoke test and every pytest batch alike. Underworld development team with AI support from Claude Code
Contributor
There was a problem hiding this comment.
Pull request overview
Pins UCX transports in the CI workflow to prevent UCX/MPICH from probing unsupported Azure MANA RDMA verbs devices that intermittently abort during MPI_Init, causing the import smoke test (and thus the whole test job) to fail before any Underworld3 code runs.
Changes:
- Adds a workflow-level
UCX_TLS: tcp,sm,selfenvironment variable to restrict UCX to TCP + shared-memory + self transports in CI. - Documents the rationale inline, referencing the observed flake pattern tracked in issues #461 and #528.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the runner-lottery flake tracked in #461 (dup #528): some Azure runners carry a MANA NIC whose InfiniBand verbs device UCX advertises but cannot open —
uct_iface_open(ud_verbs/mana_0)fails inside MPI_Init andimport underworld3dies before any UW3 code runs. Four hits in the first week of August across #466/#526/#527, each burning a manual re-run.One workflow-level env:
UCX_TLS: tcp,sm,self. A single-node CI runner needs only shared memory and TCP; the pin covers the import smoke test (where the flake fires) and every pytest batch. Local and HPC environments are untouched — this is CI-only, deliberately not in pixi.toml activation, so real InfiniBand fabrics (e.g. the Gadi work in #516) keep their transports.Underworld development team with AI support from Claude Code