Online distributed graph partitioning using PT-Scotch #1364

abishekg7 · 2025-09-24T01:27:33Z

This PR enables the use of MPAS atmosphere core without requiring any graph decomposition files specified as input, if the atmosphere core has been built with the Scotch graph partitioning library (built separately).

This PR supersedes #1348 to implement a Fortran interface to the PT-Scotch C library within MPAS.

Building MPAS with Scotch

Instructions to build Scotch are provided later in the description. The Scotch version must be at least v7.0.8. Once Scotch has been installed, the MPAS atmosphere core can be leveraged to use online graph partitioning by setting the path to Scotch prior to building MPAS.

export SCOTCH=/path/to/scotch/installation
make nvhpc CORE=atmosphere

If MPAS is built/linked with Scotch successfully, you should see a message stating that at the end of the build output.

Usage and run-time behavior

After building MPAS with Scotch, you can still choose whether or not to use online graph partitioning by setting appropriate values for the config_block_decomp_file_prefix namelist option.

If config_block_decomp_file_prefix+mpi_tasks points to a valid graph decomp file that already exists in the run directory, then it is used to proceed with the model run without invoking Scotch.
If config_block_decomp_file_prefix=='' or config_block_decomp_file_prefix+mpi_tasks does not match any valid graph decomp file in the run directory, then Scotch graph partitioning is invoked. During this process, the generated partition is saved as a graph decomp file to the run directory so that it may be reused in the following runs without needing to invoke Scotch again.

If MPAS has not been built with scotch, any code paths relating to Scotch will not be taken. And an incorrect specification of config_block_decomp_file_prefix+mpi_tasks should lead to the model halting.

Downloading and building Scotch

Scotch may be obtained from its gitlab repository or as tarballs from this link. The minimum Scotch version must be at least v7.0.8.

git clone https://gitlab.inria.fr/scotch/scotch.git && cd scotch 
git checkout v7.0.8

Build instructions are on this page

It can be as simple as

mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=prefix/scotch/install -DMPI_HOME=/your/mpi/path  ..
make -j5
make install

To be able to use the distributed graph partitioning provided by PT-Scotch, you will need to pass an appropriate MPI installation path during the Scotch build.

Other system pre-requisites are Bison and Flex. I have observed that an older Flex version (v2.6.1) on Eris caused the Scotch build to fail. After pulling in a more recent version of Flex v2.6.4 it seemed to build fine. Scotch documentation also references the requirement for a Bison version > 3.4.

PT-Scotch can be built with 64-bit integers or 32-bit integers. This is determined from build-time options and the generic integer type SCOTCH_Num covers either option. This commit replaces the type of certain integer variables, in the MPAS interface to PT-Scotch, to use SCOTCH_Num to remain consistent with the external library. Note that MPAS doesn't yet support 64-bit integers, and variables of type SCOTCH_Num are typecast into 32-bit integers in framework/mpas_block_decomp.F. And in order for MPAS to correctly work with PT-Scotch, PT-Scotch must not be built to use 64-bit integers.

…ount

src/core_atmosphere/Registry.xml

Makefile

src/framework/mpas_ptscotch_interface.F

src/framework/mpas_bootstrapping.F

src/framework/mpas_ptscotch_interface.F

Moving Scotch logic from mpas_block_decomp_cells_for_proc into a new subroutine mpas_block_decomp_scotch, in order to simplify the diff. In doing so, it also introduces some changes to logic. If mpas_block_decomp_cells_for_proc cannot find a suitable partition file AND the MPAS core has been built with Scotch, then mpas_block_decomp_scotch is invoked. If successful, this writes out a graph partition file which mpas_block_decomp_cells_for_proc then reads from disk again.

src/framework/mpas_ptscotch_interface.F

src/framework/mpas_block_decomp.F

…tran test programs

…refix is empty.

mgduda · 2026-01-15T23:38:50Z

src/framework/mpas_block_decomp.F

+
+      use mpas_timer, only : mpas_timer_start, mpas_timer_stop
+#ifdef MPAS_USE_MPI_F08
+      use mpi_f08, only : MPI_Comm, MPI_COMM_WORLD, MPI_INTEGER, MPI_Comm_dup, MPI_Comm_free, MPI_Gather, MPI_Gatherv


It doesn't look like MPI_COMM_WORLD is needed in this use statement.

mgduda · 2026-01-15T23:40:08Z

src/framework/mpas_block_decomp.F

+      integer, dimension(:), allocatable :: local_block_list
+
+      integer, dimension(:), allocatable ::global_block_id_arr, local_block_id_arr
+      integer :: i, global_block_id, local_block_id, iunit, ounit, istatus, ostatus, j, k


With the -Wall flag, the gfortran compiler notes some unused variables:

mpas_block_decomp.F:375:35: 375 | integer :: i, global_block_id, local_block_id, iunit, ounit, istatus, ostatus, j, k | 1 Warning: Unused variable 'global_block_id' declared at (1) [-Wunused-variable] mpas_block_decomp.F:375:74: 375 | integer :: i, global_block_id, local_block_id, iunit, ounit, istatus, ostatus, j, k | 1 Warning: Unused variable 'istatus' declared at (1) [-Wunused-variable] mpas_block_decomp.F:375:58: 375 | integer :: i, global_block_id, local_block_id, iunit, ounit, istatus, ostatus, j, k | 1 Warning: Unused variable 'iunit' declared at (1) [-Wunused-variable] mpas_block_decomp.F:375:51: 375 | integer :: i, global_block_id, local_block_id, iunit, ounit, istatus, ostatus, j, k | 1 Warning: Unused variable 'local_block_id' declared at (1) [-Wunused-variable] mpas_block_decomp.F:371:59: 371 | integer, dimension(:), allocatable :: local_cell_list | 1 Warning: Unused variable 'local_cell_list' declared at (1) [-Wunused-variable] mpas_block_decomp.F:379:36: 379 | character (len=StrKIND) :: msg | 1 Warning: Unused variable 'msg' declared at (1) [-Wunused-variable] mpas_block_decomp.F:385:54: 385 | doubleprecision :: scotchgraph (SCOTCH_GRAPHDIM) | 1 Warning: Unused variable 'scotchgraph' declared at (1) [-Wunused-variable]

Thanks for the catch. Fixed.

mgduda · 2026-01-15T23:45:35Z

src/framework/ptscotch_interface.c

+{
+	SCOTCH_stratExit((SCOTCH_Strat *)strat_ptr);
+}
+#endif


It looks like this file is missing and EOF character.

mgduda · 2026-01-15T23:46:22Z

src/framework/mpas_ptscotch_interface.F

+    end subroutine scotch_dgraphdata
+
+end module mpas_ptscotch_interface
+#endif


Could you add an EOF character to this file?

mgduda · 2026-01-15T23:57:03Z

src/framework/mpas_ptscotch_interface.F

+        integer(SCOTCH_NUMSIZE), intent(out) :: local_cell_list(*)
+
+        interface
+            subroutine scotchfdgraphdata(dgraph_ptr, cell_list) bind(C, name='scotchm_dgraphdata')


Here and elsewhere in this file, would it make the code easier to follow if we declared these interface names to match the C names? Or is there a reason for using interface names that differ from the names of the C routines that we're calling?

I wasn't entirely sure if that was allowed. But I've fixed this now.

mgduda · 2026-01-16T00:15:39Z

src/framework/mpas_block_decomp.F

+      endif
+      ! Initialize the Scotch graph data structure, and an extra one to hold the re-distributed graph
+#ifdef MPAS_USE_MPI_F08
+      call scotch_dgraphinit(scotchdgraph(1), localcomm% mpi_val)


Here and in other calls, it would be more in line with the rest of MPAS to pass array arguments simply using their names rather than by passing their first element.

mgduda · 2026-01-16T00:20:31Z

src/framework/mpas_ptscotch_interface.F

+
+        implicit none
+        ! Arguments
+        doubleprecision, target, intent(in) :: dgraph (SCOTCH_DGRAPHDIM)


Stylistically, I think it would be preferable to declare arrays using the dimension attribute. E.g.,

doubleprecision, dimension(SCOTCH_DGRAPHDIM), target, intent(in) :: dgraph

mgduda · 2026-01-16T00:36:04Z

src/framework/ptscotch_interface.c

+void scotchm_dgraphdata(void *ptr, SCOTCH_Num *cell_list)
+{
+
+	int err;


The err variable doesn't appear to be used.

mgduda · 2026-01-16T00:36:43Z

src/framework/ptscotch_interface.c

+	SCOTCH_Num *vendloctab = vertloctab_1 + 1;
+	SCOTCH_Num *edgeloctab = (SCOTCH_Num *)adjncy;
+
+	int i, err;


The i variable doesn't appear to be used anywhere in this function.

mgduda · 2026-01-16T00:38:11Z

src/framework/ptscotch_interface.c

+{
+	MPI_Comm comm;
+	MPI_Comm comm2;
+	int size, rank, err;


With the -Wall flag, the gcc compiler reports several unused variables:

ptscotch_interface.c: In function 'scotchm_dgraphinit': ptscotch_interface.c:37:19: warning: unused variable 'rank' [-Wunused-variable] 37 | int size, rank, err; | ^~~~ ptscotch_interface.c:37:13: warning: unused variable 'size' [-Wunused-variable] 37 | int size, rank, err; | ^~~~ ptscotch_interface.c:36:18: warning: unused variable 'comm2' [-Wunused-variable] 36 | MPI_Comm comm2; | ^~~~~

abishekg7 added 17 commits July 23, 2025 13:54

Working with upto 24km global mesh

809765e

Now checking for zero connectivities. Works with limited area grid

d46fceb

Modifying the logic for when Scotch partitioning is active

86ecb30

WIP: Makefile changes to enable SCOTCH

2b6258c

Adding compile-time link test for the Scotch library

d677fb4

few fixes for GNU and Intel builds

2176657

adding timers

d63accc

PT-Scotch WIP

ebbd919

working pt-scotch

d8207e6

removing inq of unneeded vars

d4f3f2d

cleanup prior to adding MPAS_SCOTCH logic

6d2d9fd

Scotch is selectively enabled when SCOTCH is set

fcd89ef

More cleanup and added comments

52c7830

Writing out correct graph parition files from PT-Scotch

a073471

WIP

d04cbf8

Working interfaces

4e105d4

cleanup

f721710

mgduda mentioned this pull request Oct 16, 2025

Online graph partitioning using Scotch #1339

Closed

abishekg7 added 4 commits October 22, 2025 10:45

mpas_block_decomp.o depends on mpas_ptscotch_interface.o

d3f511c

introducing some new variables

67c25ba

Using SCOTCH_STRATSCALABILITY to avoid crashes with large partition c…

029d3e1

…ount

mgduda reviewed Dec 1, 2025

View reviewed changes

src/core_atmosphere/Registry.xml Outdated Show resolved Hide resolved

mgduda reviewed Dec 1, 2025

View reviewed changes

Makefile Outdated Show resolved Hide resolved

mgduda reviewed Dec 1, 2025

View reviewed changes

src/framework/mpas_ptscotch_interface.F Outdated Show resolved Hide resolved

mgduda reviewed Dec 1, 2025

View reviewed changes

src/framework/mpas_bootstrapping.F Outdated Show resolved Hide resolved

mgduda reviewed Dec 1, 2025

View reviewed changes

src/framework/mpas_ptscotch_interface.F Show resolved Hide resolved

abishekg7 added 3 commits December 6, 2025 08:49

Undoing changes to config_block_decomp_file_prefix defaults

103e8a5

Make sure non-Scotch build still works

0eee657

abishekg7 added 12 commits December 6, 2025 09:17

Removing timers from mpas_bootstrapping

7accc40

Cleaning up whitespace

697955f

Apply a more consistent formatting

ff4e012

Adding comments and removing return for void calls

38d6a86

Adding some documentation

602c79a

Fixes and cleanup

2188fef

Removing DGraph2 and using SCOTCH_dgraphData, SCOTCH_dgraphSize instead

82450f2

Fixes and cleanup

f92997e

Cleaning up whitespace and variables in mpas_block_decomp_scotch

8849a0b

Prevent scotch logic when config_number_of_blocks is not zero

87c7358

Cleanup and converting Scotch F interface functions to subroutines

be6cfa2

Replace total_blocks with nProcs

7d370ad

mgduda reviewed Dec 23, 2025

View reviewed changes

src/framework/mpas_ptscotch_interface.F Show resolved Hide resolved

mgduda requested changes Dec 23, 2025

View reviewed changes

src/framework/mpas_block_decomp.F Outdated Show resolved Hide resolved

src/framework/mpas_block_decomp.F Outdated Show resolved Hide resolved

src/framework/mpas_block_decomp.F Outdated Show resolved Hide resolved

abishekg7 added 7 commits December 26, 2025 15:07

Some minor edits to address review comments

79f759a

Replacing MPI_COMM_WORLD with dminfo%comm

3150b1a

Fix to non-Scotch code path

ef465d4

Updating Scotch build tests in Makefile. Now tries to build C and For…

6cd6eb9

…tran test programs

Updating copyright block

0ef7da0

Small fixes and cleanup for code path when config_block_decomp_file_p…

47d7701

…refix is empty.

Cleanup of print statements

bc93fb0

abishekg7 marked this pull request as ready for review December 29, 2025 18:11

abishekg7 requested a review from mgduda December 29, 2025 18:11

mgduda requested changes Jan 16, 2026

View reviewed changes

abishekg7 added 4 commits January 16, 2026 16:49

Removing unused variables and adding EOF as needed

2acccc8

Declaring arrays using the dimension attribute

13ff186

Addressing some review comments

d400e4f

Changing Fortran interface names to match C function names

e7b6094

Online distributed graph partitioning using PT-Scotch #1364

Are you sure you want to change the base?

Online distributed graph partitioning using PT-Scotch #1364

Conversation

abishekg7 commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Building MPAS with Scotch

Usage and run-time behavior

Downloading and building Scotch

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

abishekg7 commented Sep 24, 2025 •

edited

Loading