Scalable Hybrid Adaptive Mesh Refinement: Towards End-to-end Simulation (Hybrid AMR Simulation)

• Applicants: Professor Dr. Carsten Burstedde; Dr. Johannes Holke
• Subject Area: Computer Architecture, Embedded and Massively Parallel Systems ; Mathematics
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 467255783

In this project we aim to continue our research on massively parallel tree-based adaptive mesh refinement (AMR) algorithms for arbitrary element shapes with particular focus on applications in earth system modelling.In the recent decades tree-based AMR has proven to be a reliable technique to improve the performance and scalability of numerical simulation codes. For 3D simulations, however, these techniques were until recently limited to hexahedral shapes. In our own research since 2015, we have been developing mathematical theories and algorithms to extend 3D tree-based AMR to all commonly used element shapes, in particular tetrahedra, hexahedra, prisms and pyramids. These are implemented in the t8code software library. We demonstrated scalability of our AMR algorithms on up to 917k parallel processes and up to 1.1e12 elements with runtimes on the order of one second wall clock time. Currently we are working with application codes in the earth system modelling community that would benefit from flexible and fast adaptive meshing.For this proposal, our first goal is to improve and further develop central tree-based AMR algorithms in order to provide fast and scalable algorithms supporting a complete set of capabilities as expected of a leading tree-based AMR library. In particular, we plan to develop a mesh-global parallel search algorithm, a state-of-the-art 2:1 balance routine, and routines for parallel I/O of tree-based meshes and associated simulation data.Our second goal is to design and implement an API to call from external codes in order to simplify the integration of t8code’s capabilities into existing simulation software. On the one hand, this would include an abstract iterate functionality to efficiently traverse the mesh from any application. On the other hand, we will interface to the widely used scientific computing package PETSc to leverage its discretization and solver support.Going beyond the scope of a software library and its API, we will develop a standalone application in the context of earth system modelling (ESM) inside the MESSy framework in order to benchmark all newly researched algorithms and to demonstrate their unique capabilities. We plan to scale both our individual technical algorithms and this application as a whole to the full system size of the Juwels supercomputer at the Jülich Supercomputing Center, Germany.Our efforts will allow modellers and researchers to quickly set up a numerical simulation leveraging scalable and dynamic tree-based AMR and to migrate their existing codes from using unstructured meshes to using tree-based meshes. Adopting t8code as low-level meshing functionality, the performance of third-party codes can be elevated through inheriting increased scalability, decreased memory footprint, powerful mesh interrogation features and overall reduced runtimes.

• DFG Programme: Research Grants
• Co-Investigator: Dr.-Ing. Achim Basermann