Skip to content

Algorithms over pieces take MPI ranks that hold no pieces - #365

Merged
berkgeveci merged 1 commit into
Kitware:mainfrom
berkgeveci:mpi/empty-ranks
Nov 5, 2026
Merged

berkgeveci merged 1 commit into
Kitware:mainfrom
berkgeveci:mpi/empty-ranks

Conversation

@berkgeveci

Copy link
Copy Markdown
Collaborator

A reader that gives some ranks nothing leaves those ranks without pieces. Every rank must make the same communicating calls, so no rank may decide from its own pieces alone what the others decide from theirs. Algorithms that did either refused on such a rank or hung the job.

Fixed

  • Refused or failed on a rank without pieces, leaving the others waiting: probe of pieces or an AMR hierarchy, streamlines and advect over pieces, voxelize, and Advection (with the path filters). Such a rank now takes part through a piece of nothing (_common._stand_in): its points and seeds go to the pieces that hold them, and the answers come back.
  • Chose their fields, or a shortcut, from this rank's pieces: update_ghosts, extract_surface of pieces, AMR ghost_cells and average_down, integrate over pieces, and the VTK Statistics, Entropy and Integrate algorithms. A rank without any now gets the others' choice. The VTK Integrate also takes one origin for the centroid over every rank.
  • merge_pieces onto a root rank without pieces refused. The merge is now made on the first rank that has pieces and moved to the root.

Test

test_empty_ranks.py runs _empty_ranks_mpi.py under mpiexec -n 3. It has 36 cases: pieces, AMR hierarchies, octrees, advection and the VTK algorithms over pieces. They run with four pieces on rank 0 only, on ranks 1 and 2, on ranks 0 and 1, and on none. Each answer is held to the same pieces in one process.

  • Hangs are stopped by a deadline that names where each rank was.
  • No pieces anywhere may refuse, with the same error on every rank.
  • CI: the MPI job runs the test as a step of its own. Its limit goes from 15 to 30 minutes; the test takes about 1-1.5 minutes a layout here.

Not fixed: an error only the ranks with data can detect (a scalar field given as a velocity) still leaves the others waiting (#364, 0.8).

Tests: test_empty_ranks.py (4 layouts), the touched modules' tests with their MPI ones (672 passed), the quick tier (3,124), lint.

🤖 Generated with Claude Code

A reader that gives some ranks nothing makes ranks without pieces. Each rank
must make the same calls that communicate, so none may decide from its own
pieces alone what the others decide from theirs.

- probe of pieces or an AMR hierarchy, streamlines and advect over pieces,
  voxelize, and Advection (with the path filters) refused or failed on a rank
  without pieces, leaving the others waiting. Such a rank now takes part
  through a piece of nothing (_common._stand_in): its points and seeds go to
  the pieces that hold them and the answers come back.
- update_ghosts, extract_surface of pieces, AMR ghost_cells and
  average_down, integrate over pieces, and the VTK Statistics, Entropy and
  Integrate algorithms chose their fields, or a shortcut, from this rank's
  pieces; a rank without any now gets the others' choice.
- merge_pieces onto a root rank without pieces is made on the first rank
  that has pieces and moved to the root.

test_empty_ranks.py runs 36 cases (pieces, AMR hierarchies, octrees,
advection, VTK algorithms) on 3 ranks with the pieces on one, two or none of
them, against the same pieces in one process; CI's MPI job runs it.
@berkgeveci
berkgeveci merged commit 50c183e into Kitware:main Nov 5, 2026
30 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant