Repository navigation
Block-structured adaptive mesh refinement (AMR) - #2117
Draft
sbryngelson wants to merge 27 commits into
Draft
sbryngelson wants to merge 27 commits into
sbryngelson wants to merge 27 commits into
Conversation
…ecking, Open MPI oversubscribe, stage-guard lint, validator fixes format.sh treats ffmt's unmatched-structure warnings as fatal; Cray debug builds get -h bounds like the other compilers; the run template passes --oversubscribe to Open MPI when ranks exceed cores; a source lint rejects per-executable preprocessor guards; the validator derives cfl_dt like the Fortran, rejects MHD with Euler bubbles and non-positive rburn%pign.
On amdflang OpenMP offload a private local array is mapped tofrom on every launch; privateArrays maps it alloc instead (OpenACC merges it into private). Opt-in present:allocatable per source file.
…ilds compile only the selected Riemann solver Stiffened-gas builds drop the state-dependent EOS chain from every kernel. CI builds a third simulation variant for the Mie-Gruneisen, JWL and Vinet cases, and a slug check guards the case-optimization pre-build.
…L/HLLC/LF/HLLD, copy primitives only for readers The Riemann solvers write module flat buffers that CBC, QBMM, surface tension, conduction and chemistry read directly (flux_n and flux_gsrc_n are gone). The reconstruction input is packed once per RHS. HLLC face work moves to routines with explicit-shape arrays. q_prim_vf is copied out only for its readers (probes, IB, Lagrangian bubbles, synthetic turbulence, run_time_info); hyper_cleaning's psi is always copied. Results are unchanged: every master golden passes.
|
Claude Code Review Head SHA: f08f835 Files changed:
Findings:
Not reviewed in depth: the bulk of the |
…packed-input reuse comment
… default tolerance
…and case-optimization guards
…HLLD and hypo HLLD
…and Re_avg to -1:m like flux_rsx_vf
…RALLEL_LOOP takes privateArrays; document MFC_OMP_PRESENT_ALLOCATABLE
…; fix the case.py PROHIBIT reference
…ents, unused s_lag_cloud_bbox_local and WENO public entry; fix WENO stencil comment
…eholder allocations and buffer zeroing out in place
…guards; emit privateArrays as its own private clause instead of a list-joining helper
…ylindrical viscous overran -1:m)
…rmat on ffmt structure warnings; Cray debug bounds checking; Open MPI oversubscribe; stage-guard lint; validator fixes bc_type is allocated (1:num_dims, 1:2) but simulation's m_data_output declared it (1:num_dims, -1:1). Serial readers need bc_type.dat and bc_buffers.dat in every step directory. format.sh treats ffmt's unmatched-structure warnings as fatal; Cray debug builds get -h bounds like the other compilers; the run template passes --oversubscribe to Open MPI when ranks exceed cores; a source lint rejects per-executable preprocessor guards; the validator derives cfl_dt like the Fortran and rejects MHD with Euler bubbles and a non-positive rburn%pign.
…ry helpers, restart I/O
…M, IGR, physics, startup, output)
…iction/reflux, clustering, nesting, regrid, subcycling, restart)
Lines of Code
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2116 (solver performance) and #2118 (small fixes). Until they merge this diff includes their commits;
the AMR changes are the last 8 commits.
Contribution Policy
All contributions are expected to demonstrate:
Summary
Adds block-structured adaptive mesh refinement to
simulation, off by default (amr = F). With AMR off, results arethose of #2116 (every existing golden passes unchanged).
amr_max_grid_sizecells, refinement ratio 2 (or 4 for one level), up to
amr_max_levelproperly nested levels, regrid everyamr_regrid_intsteps.amr_subcycle, two fine steps per coarse step, up to two refinedlevels) with time-interpolated ghosts. Flux registers reflux every coarse/fine face, so the scheme is conservative.
amr_prolong = 2) with a fallback to linearwherever the quartic state would have non-positive density or internal energy; conservative restriction.
with
amr_place_owner). Exchanges are nonblocking waves with keyed tags, so results do not depend on message timing.Lagrangian bubbles (lock-step, cloud on the base grid), acoustic sources (base grid), 1D MHD, IGR (single rank), and
immersed boundaries (fixed, prescribed or force-driven motion).
docs/documentation/amr.md(user guide),amr_implementation.md(internals), one AMR section incase.md.The solver runs on a block by swapping its grid globals (
m,n,p, index ranges, coordinates) in and out;amr_implementation.mdstates the contract for module state that depends on the grid. AMR validation fails closed: withamr = Tthe validator rejects any logical switch not inAMR_VETTED_FLAGS, so new physics is refused until it meets thatcontract. Explicitly rejected with a message:
model_eqns = 1, Euler-Euler bubbles, QBMM, phase change, surface tension,synthetic turbulence, cylindrical or stretched grids,
bc = -4withnull_weights; subcycling with IGR, chemistry,Lagrangian bubbles or moving bodies.
Verification
Regression
-c frontier_amd, 4 shards)--no-mpi)CCE found two local allocatable arrays named in kernel data clauses that its OpenMP runtime looked up in the present table
anyway; both are now automatic arrays.
New goldens: the AMR cases (52 tests: multi-rank, multi-level, subcycling, restart, IB), quartic prolongation (1D, and 2D
with IB), and
rank_time_wrtdiagnostics. No existing golden changes.Accuracy and conservation
1D periodic Euler, WENO5, SSP-RK3, refinement ratio 2:
L1 density error after one period. Third order is kept; the error constant is larger because the coarse/fine ghost fill is
second order. Total mass, momentum and energy change by 1.4e-16, 4.2e-16 and 1.9e-16 (relative) over 200 steps with regrid
every 4 steps. The same run on 1 and 4 ranks is bitwise identical.
Performance (weak scaling, blast lattice, 8M base cells per GPU, one refined level, subcycled)
s/step (efficiency vs 8 GPUs):
MFC refines about 1.3x more volume than AMReX and PeleC here (40.5% vs 30.3%). The table is from an earlier commit of this
work; the 64-GPU rung rerun before the final cleanup gave 0.680 s/step against 0.698 for that commit, and a rerun on the
final tree is pending. Regrid cost grows faster than the step beyond 512 GPUs; that is a follow-up.
Use
Mach 2.5 shock through 297 fixed random spheres: 384 x 256 x 256 base grid plus one level, 128 GPUs, 1350 steps, up to
about 5300 blocks.
Known limitations and follow-ups
Regrid keeps the current blocks when no cell is tagged, so refinement does not retract until something is tagged again.
Level-2 and deeper blocks are re-prolonged from their parent at every regrid (only level-1 data is carried over or
migrated), which costs accuracy in multi-level runs that regrid often.
The AMR restart header records the refinement ratio, maximum level and word size and is checked on read.
Regrid cost above 512 GPUs.
The solver reads grid state from module globals; AMR swaps them per block. Passing the grid explicitly would remove
the swap but touches the whole solver.
m_amr_stateholds many module variables; a block-table derived type would be cleaner.