Skip to content

CI flake: MPI_Init dies in UCX on some Azure runners (mana_0 verbs transport) #528

Description

@lmoresi

Twice in two days (PR #526 first run, PR #527 first run) the test job died at the import smoke test, before any underworld3 code loaded:

ib_iface.c:1316 UCX ERROR mana_0: iface ... failed to create UD QP ... Operation not supported
ucp_worker.c:1415 UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Input/output error
MPIDI_UCX_init_worker(86): ucx function returned with failed status

UCX is trying an unsupported verbs transport on the runner's Azure MANA network interface; whether a run flakes depends on which VM type the job lands on. Rerun on a fresh runner passes (measured, #526).

Suggested immunisation: pin the transport in the workflow env for the test job, e.g.

env:
  UCX_TLS: tcp,self,sm

(or UCX_NET_DEVICES=lo for the serial import smoke alone). Filed so the reruns stop being folklore.

Underworld development team with AI support from Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions