Introduction to PETSc Data management Loïc Gouarin Laboratoire de Mathématiques d Orsay May 13-15, 2013 Loïc Gouarin Introduction to PETSc 1
Data management PETSc offers two types of data management DMDA: data management for structured mesh DMPlex (or DMMesh): data management for unstructured mesh These structures define for each process local portion of the mesh, ghost points, communications with the neighbourhood to update ghost points, global and local mapping,... Loïc Gouarin Introduction to PETSc 2
DMDA creation int DMDACreate2d(MPI_Comm comm, DMDABoundaryType xperiod, DMDABoundaryType yperiod, DMDAStencilType st, int M, int N,int m,int n,int dof,int s, int *lx,int *ly,dm *da) xperiod and yperiod: type of ghost nodes. DMDA BOUNDARY NONE, DMDA BOUNDARY GHOSTED, DMDA BOUNDARY PERIODIC st: stencil type. DMDA STENCIL BOX or DMDA STENCIL STAR M and N: global dimension in each direction. m and n: number of processors in each direction. dof: number of degrees of freedom per node. s: stencil width. Loïc Gouarin Introduction to PETSc 3
Local and global vectors Creation int DMCreateGlobalVector(DM da,vec *g) int DMCreateLocalVector(DM da,vec *l) Scatter a global vector into its local parts including the ghost points DMGlobalToLocalBegin(DM da,vec g, InsertMode iora,vec l); DMGlobalToLocalEnd(DM da,vec g, InsertMode iora,vec l); Scatter a local vector into the global vector DMDALocalToGlobalBegin(DM da,vec l, InsertMode iora,vec g); DMDALocalToGlobalEnd(DM da,vec l, InsertMode iora,vec g); InsertMode can be either INSERT_VALUES or ADD_VALUES. Loïc Gouarin Introduction to PETSc 4
First example #include "petsc.h" int main(int argc, char **argv){ int nx=5, ny=5; DM dm; Vec g; } PetscInitialize(&argc, &argv, NULL, NULL); DMDACreate2d(PETSC_COMM_WORLD, DMDA_BOUNDARY_NONE, DMDA_BOUNDARY_NONE, DMDA_STENCIL_STAR, nx, ny, PETSC_DECIDE, PETSC_DECIDE, 1, 1, PETSC_NULL, PETSC_NULL, &dm); DMCreateGlobalVector(dm, &g);... VecDestroy(&g); PetscFinalize(); return 0; Loïc Gouarin Introduction to PETSc 5
Local portion on each process 2 3 0 1 Loïc Gouarin Introduction to PETSc 6
Local portion on each process 20 21 22 23 24 15 16 17 18 19 10 11 12 13 14 5 6 7 8 9 0 1 2 3 4 Loïc Gouarin Introduction to PETSc 7
Ghost points: DMDA STENCIL STAR 0 Loïc Gouarin Introduction to PETSc 8
Ghost points: DMDA STENCIL STAR 3 Loïc Gouarin Introduction to PETSc 9
Ghost points: DMDA STENCIL BOX 0 Loïc Gouarin Introduction to PETSc 10
Ghost points: DMDA STENCIL BOX 3 Loïc Gouarin Introduction to PETSc 11
How to get grid information? DMDAGetCorners(DM da, int *xs,int *ys,int *zs, int *xm,int *ym,int *zm); ym 0 (xs, ys) xm Use PETSC_NULL if you want to omit a parameter. Loïc Gouarin Introduction to PETSc 12
How to get grid information? DMDAGetCorners(DM da, int *xs,int *ys,int *zs, int *xm,int *ym,int *zm); ym 3 (xs, ys) xm Loïc Gouarin Introduction to PETSc 13
How to get grid information? DMDAGetGhostCorners(DM da, int *gxs,int *gys,int *gzs, int *gxm,int *gym,int *gzm); gym 0 (gxs, gys) gxm Loïc Gouarin Introduction to PETSc 14
How to get grid information? DMDAGetGhostCorners(DM da, int *gxs,int *gys,int *gzs, int *gxm,int *gym,int *gzm); gym 3 (gxs, gys) gxm Loïc Gouarin Introduction to PETSc 15
How to get grid information? DMDAGetLocalInfo(DM da,dmdalocalinfo *info) typedef struct { PetscInt dim,dof,sw; /* global number of grid points in each direction */ PetscInt mx,my,mz; /* starting point of this processor, excluding ghosts */ PetscInt xs,ys,zs; /* number of grid points on this processor, excluding ghosts * PetscInt xm,ym,zm; /* starting point of this processor including ghosts */ PetscInt gxs,gys,gzs; /* number of grid points on this processor including ghosts */ PetscInt gxm,gym,gzm; /* type of ghost nodes at boundary */ DMDABoundaryType bx,by,bz; DMDAStencilType st; DM da; } DMDALocalInfo; Loïc Gouarin Introduction to PETSc 16
DMDA offers functions for vector manipulation Local (ghosted) work vectors DMGetLocalVector(DM da,vec *l);... use the local vector l DMRestoreLocalVector(DM da,vec *l); Accessing the vector entries for DMDA vectors PetscScalar **f,**u;... DMDAVecGetArray(DM da,vec local,&u); DMDAVecGetArray(DM da,vec global,&f);... f[i][j] = u[i][j] -...... DMDAVecRestoreArray(DM da,vec local,&u); DMDAVecRestoreArray(DM da,vec global,&f); Loïc Gouarin Introduction to PETSc 17
References 1 PETSc documentation http://www.mcs.anl.gov/petsc/documentation/index.html 2 PETSc tutorial http://www.mcs.anl.gov/petsc/documentation/tutorials/index.html Loïc Gouarin Introduction to PETSc 18