ParFUM: A Parallel Framework for Unstructured Meshes. Aaron Becker, Isaac Dooley, Terry Wilmarth, Sayantan Chakravorty Charm++ Workshop 2008

Transcription

1 ParFUM: A Parallel Framework for Unstructured Meshes Aaron Becker, Isaac Dooley, Terry Wilmarth, Sayantan Chakravorty Charm++ Workshop 2008

2 What is ParFUM? A framework for writing parallel finite element codes Takes care of difficult tasks involved in parallelizing a serial code Provides advanced mesh operations such as mesh adaptivity and dynamic load balancing Constantly evolving to support application needs (for example, cohesive elements and collision detection) Based on Charm++ and AMPI. Supports C, C++, and Fortran

3 Making Parallel Finite Element Codes Easier A simple serial finite element code: Create mesh Perform finite element computations Extract results

4 Making Parallel Finite Element Codes Easier A simple parallel finite element code: Create mesh Partition mesh Distribute mesh data and create ghost layers Perform finite element computations with synchronization and load balancing Extract results

5 Making Parallel Finite Element Codes Easier Create mesh ParFUM can do these things automatically and let the developer concentrate on science and engineering Partition mesh Distribute mesh data and create ghost layers Perform finite element computations with synchronization and load balancing Extract results

6 The Structure of a ParFUM Program Init MP Cha MS Partitioning and Distribution Driver

7 The Big Picture Adjacency Generation User's Solver ParFUM Solution Transfer Bulk Adaptivity Ghost Layer Generation Partitioning IFEM ptops Collision Detection Contact Incremental Adaptivity Multi-phase Shared Arrays User View AMPI Charm++ System View Load Balancing Framework Charm Run-time System Communication Optimizations 7

8 Integrating Multiple Programming Models Adjacency Generation Global Shared Memory Ghost Layer Generation Partitioning User's Solver ParFUM Message ptops Passing IFEM Solution Transfer Collision Detection Message Contact Driven Bulk Adaptivity Incremental Adaptivity Multi-phase Shared Arrays User View AMPI Charm++ System View Load Balancing Framework Charm Run-time System Communication Optimizations 8

9 ParFUM and AMPI Application code is written in AMPI, an implementation of MPI on top of the Charm RTS. AMPI processes (virtual processors, or VPs) are not tied to a physical processor, they can migrate and there may be many of them per physical processor This allows easier porting of MPI codes and eases the learning curve of ParFUM

10 Virtualization Tradeoffs Advantages Allows adaptive overlap of communication and computation More granular load balancing Improved cache performance More flexibility Disadvantages More communication Worse ratio of remote data to local data Imposes some thread overhead High virtualization requires many elements per node

11 Virtualization Performance Impact For this dynamic fracture code, virtualization provides a substantial benefit

12 Parallel Mesh Adaptivity Efficient parallel adaptivity is critical for many unstructured mesh codes ParFUM provides two implementations of common operations: incremental (2D triangle meshes): each individual operation leaves the mesh consistent. Relatively slow and puts limitations on ghost layers bulk (2D triangles and 3D tets): many operations performed at once, ghosts and adjacencies updated at end. Lower cost, no restrictions on ghost layers. (ongoing work)

13 Higher Level Adaptivity Propagating Edge Bisection Operations like propagating edge bisection are composed from edge bisect, flip, and contraction primitives Which is better, bulk or incremental? Depends on amount and frequency of adaptivity

14 Load Balance, Adaptivity, and Virtualization Serious load imbalance: areas near fracture are much more expensive

15 Load Balance, Adaptivity, and Virtualization We can change the VP mapping to distribute computationally expensive parts of the mesh better

16 Load Balance, Adaptivity, and Virtualization Assigning VPs using a greedy load balancer further improves utilization

17 Spacetime Meshing Parallelization of Spacetime Discontinuous Galerkin (SDG) algorithm [Haber] Adaptive in both space and time, uses incremental adaptivity Asynchronous code, no global barriers

18 PTops Structural dynamics code for graded materials [Paulino] Based on Tops, a serial framework featuring an efficient topological mesh representation

19 PTops Strong Scaling 400,000 elements on Abe No virtualization

20 PTops and CUDA ParFUM-Tops interface has CUDA support Our implementation runs ~10x faster on a single node using CUDA Limited usefulness due to lack of double precision and lack of access to clusters which combine GPUs and high quality interconnects

21 Ongoing Work On-demand insertion of cohesive elements (truly extrinsic cohesives) in PTops for dynamic fracture simulations Efficient, scalable implementation of bulk edge flip and edge contraction Contact: use Charm++ collision detection library to detect when domain fragments come into contact

22 ParFUM: A Parallel Framework for Unstructured Meshes Aaron Becker, Sayantan Chakravorty, Isaac Dooley, Terry Wilmarth Parallel Programming Lab Charm++ Workshop 2008